6 ms·

If you bother to write a python script to parse the integers, why not use python to solve the whole problem?

by yuriko 8y ago

ekianjo 8y ago
yeah that was super weird. Why write an article about the merits of shell tools if you put some python in the mix...
- Retra 8y ago
  Python basically acts like a subshell with its own language... I don't see why anyone would think unix shell scripts are really that different from Python scripts, especially if you're not doing the subprocess control things that command shells are optimal for. Invoking python to do something doesn't seem any more awkward to me than invoking sed, awk, etc..
  - ekianjo 8y ago
    Python has no bindings to the underlying OS (unless you import os modules, and even then it has its own way of working), and does not interact directly with environment variables either, and as such can not be seriously considered like a "shell" tool at the same level as the other ones. It just was never designed to act in this way. And in the embedded world you don't want to install a python interpreter if you can do without.
    - Retra 8y ago
      If you're just sorting numbers, you don't need the OS or environment variables.
v3gas 8y ago
Fair point! I'd argue there's a difference in time spent in writing a python script to solve it all, and just parsing the ints as I did. Python was my first thought for how to parse the ints.
- james_s_tayler 8y ago
  Great article. I didn't know about comm actually, so that was new for me! With regard to the 'ints' it helps to not think about them as ints but rather just some text that follows the pattern ^[0][1-9] or some equivalent to that. "Starts with any number of zeros or no zeros followed by any number of numbers between 1 to 9 or none at all." So long as you know the repeatable pattern you can always use sed to just replace the part of the pattern you want gone with nothing which effectively deletes it from the output. sed is like a Swiss army knife in that regard because you can do nice simple deletions like that and even iterate on them if you need or you can do quite complicated capture groups if you need to as well. Sed can get you unbelievably far in terms of shaping text in a stream. I have a few tricks I've learned with various tools that I thought were worth writing down. Hopefully you can find some more useful stuff. https://github.com/nicostouch/grep-sed-awk-magic/blob/master/README.md https://github.com/nicostouch/grep-sed-awk-magic/blob/master...
  - pbhjpbhj 8y ago
    Your regex got munged, but shouldn't it be: ^[0]*[0-9]+$ to avoid matching empty lines?
    - james_s_tayler 8y ago
      Oh I see. Yes the output got a bit messed up. ^[0]*[1-9]* You could use + if you want. I only do it if not doing will produce incorrect output for my given input, which happens occasionally but is rare. Bear in mind in this case there is also _A.csv or _B.csv you need to account for as well which your version doesn't. Mine will still pick it up because I'm not specifying the end, as I'm assuming I've done the necessary preprocessing steps to get good data so it will produce the expected output. I'm not usually fussed on being that strict when I write regex as I tend to do various preprocessing steps like grep -v ^$ to filter out any blank lines if I don't want them in the stream etc.
      - pbhjpbhj 8y ago
        Ya, I dropped in an unwanted $.
AnIdiotOnTheNet 8y ago
This is one of the many reasons I think PowerShell did UNIX philosophy better: you don't need to parse text because the pipelines pass around typed objects. You can kinda almost get the same behavior from some UNIX commands by first having them dump everything into JSON and then having the other end parse the JSON for you, but you're still relying on a lot of text parsing. Personally I think it is high time the UNIX world put together a new toolset.
- useerup 8y ago
  Indeed, in PowerShell you could do: 1..500 | ?{ !(Test-Path ('{0:0000}_A.csv' -f $_)) } Explanation: 1..500 generates a sequence of numbers 1 through 500 | pipes the numbers ?{ … } is a filter that is evaluated for each item (number) ! negates the following expression Test-Path tests that a file exists -f formats the string left of -f with the parameters (zero-based to the right of -f '{0:0000}_A.csv' is a pattern which formats parameter 0 as 4 digits, zero-padded. EDIT: Explanation
  - jclay 8y ago
    Great example. PowerShell is definitely underrated. I don’t care to use it as an interactive shell, and that likely turns many off to it, but now that it’s cross platform I think it’s massively underrated for these sorts of tasks.
    - eMSF 8y ago
      I don't think that example really shows the benefits of the "Powershell way". There's hardly any need for objects and what-not when solving an easy problem like this; strings will do just fine. With Bash, a close equivalent would be: for f in {0001..0500}_A.csv; do test -e $f || echo $f; done
- geophile 8y ago
  Take a look at osh (Object SHell): https://github.com/geophile/osh https://github.com/geophile/osh It is a Python implementation of this idea: OS objects like files and processes are represented in Python. You construct pipelines as in UNIX, but passing Python objects instead of strings. E.g. to find the pids of /bin/bash commands: osh ps ^ select 'p: p.commandline.startswith("/bin/bash")' ^ f 'p: p.pid' $ - osh: Runs the tool, interpreting the rest of the line as an osh command. - ^: Piping syntax. - ps: Generate a stream of process objects, (the currently running processes). - select: Select those processes, p, whose commandline starts with /bin/bash. - f: Apply the function to the input stream and write function output to the output stream, (so, basically "map"). The function computes the pid of an input process. - $ Print each item received in the input stream. Osh also does database access (query results -> python tuples) and remote access. E.g., to get a process listing of (pid, commandline) on every node in a cluster: osh @clustername [ ps ^ f 'p: (p.pid, p.commandline)' ] $

If you bother to write a python script to parse the integers, why not use python to solve the whole problem?

by yuriko 8y ago

If you bother to write a python script to parse the integers, why not use python to solve the whole problem?

ekianjo 8y ago

yeah that was super weird. Why write an article about the merits of shell tools if you put some python in the mix...

Retra 8y ago
Python basically acts like a subshell with its own language... I don't see why anyone would think unix shell scripts are really that different from Python scripts, especially if you're not doing the subprocess control things that command shells are optimal for. Invoking python to do something doesn't seem any more awkward to me than invoking sed, awk, etc..
- ekianjo 8y ago
  Python has no bindings to the underlying OS (unless you import os modules, and even then it has its own way of working), and does not interact directly with environment variables either, and as such can not be seriously considered like a "shell" tool at the same level as the other ones. It just was never designed to act in this way. And in the embedded world you don't want to install a python interpreter if you can do without.
  - Retra 8y ago
    If you're just sorting numbers, you don't need the OS or environment variables.

v3gas 8y ago

Fair point! I'd argue there's a difference in time spent in writing a python script to solve it all, and just parsing the ints as I did. Python was my first thought for how to parse the ints.

james_s_tayler 8y ago
Great article. I didn't know about comm actually, so that was new for me! With regard to the 'ints' it helps to not think about them as ints but rather just some text that follows the pattern ^[0][1-9] or some equivalent to that. "Starts with any number of zeros or no zeros followed by any number of numbers between 1 to 9 or none at all." So long as you know the repeatable pattern you can always use sed to just replace the part of the pattern you want gone with nothing which effectively deletes it from the output. sed is like a Swiss army knife in that regard because you can do nice simple deletions like that and even iterate on them if you need or you can do quite complicated capture groups if you need to as well. Sed can get you unbelievably far in terms of shaping text in a stream. I have a few tricks I've learned with various tools that I thought were worth writing down. Hopefully you can find some more useful stuff. https://github.com/nicostouch/grep-sed-awk-magic/blob/master/README.md https://github.com/nicostouch/grep-sed-awk-magic/blob/master...
- pbhjpbhj 8y ago
  Your regex got munged, but shouldn't it be: ^[0]*[0-9]+$ to avoid matching empty lines?
  - james_s_tayler 8y ago
    Oh I see. Yes the output got a bit messed up. ^[0]*[1-9]* You could use + if you want. I only do it if not doing will produce incorrect output for my given input, which happens occasionally but is rare. Bear in mind in this case there is also _A.csv or _B.csv you need to account for as well which your version doesn't. Mine will still pick it up because I'm not specifying the end, as I'm assuming I've done the necessary preprocessing steps to get good data so it will produce the expected output. I'm not usually fussed on being that strict when I write regex as I tend to do various preprocessing steps like grep -v ^$ to filter out any blank lines if I don't want them in the stream etc.
    - pbhjpbhj 8y ago
      Ya, I dropped in an unwanted $.

AnIdiotOnTheNet 8y ago

This is one of the many reasons I think PowerShell did UNIX philosophy better: you don't need to parse text because the pipelines pass around typed objects. You can kinda almost get the same behavior from some UNIX commands by first having them dump everything into JSON and then having the other end parse the JSON for you, but you're still relying on a lot of text parsing. Personally I think it is high time the UNIX world put together a new toolset.

useerup 8y ago
Indeed, in PowerShell you could do: 1..500 | ?{ !(Test-Path ('{0:0000}_A.csv' -f $_)) } Explanation: 1..500 generates a sequence of numbers 1 through 500 | pipes the numbers ?{ … } is a filter that is evaluated for each item (number) ! negates the following expression Test-Path tests that a file exists -f formats the string left of -f with the parameters (zero-based to the right of -f '{0:0000}_A.csv' is a pattern which formats parameter 0 as 4 digits, zero-padded. EDIT: Explanation
- jclay 8y ago
  Great example. PowerShell is definitely underrated. I don’t care to use it as an interactive shell, and that likely turns many off to it, but now that it’s cross platform I think it’s massively underrated for these sorts of tasks.
  - eMSF 8y ago
    I don't think that example really shows the benefits of the "Powershell way". There's hardly any need for objects and what-not when solving an easy problem like this; strings will do just fine. With Bash, a close equivalent would be: for f in {0001..0500}_A.csv; do test -e $f || echo $f; done
geophile 8y ago
Take a look at osh (Object SHell): https://github.com/geophile/osh https://github.com/geophile/osh It is a Python implementation of this idea: OS objects like files and processes are represented in Python. You construct pipelines as in UNIX, but passing Python objects instead of strings. E.g. to find the pids of /bin/bash commands: osh ps ^ select 'p: p.commandline.startswith("/bin/bash")' ^ f 'p: p.pid' $ - osh: Runs the tool, interpreting the rest of the line as an osh command. - ^: Piping syntax. - ps: Generate a stream of process objects, (the currently running processes). - select: Select those processes, p, whose commandline starts with /bin/bash. - f: Apply the function to the input stream and write function output to the output stream, (so, basically "map"). The function computes the pid of an input process. - $ Print each item received in the input stream. Osh also does database access (query results -> python tuples) and remote access. E.g., to get a process listing of (pid, commandline) on every node in a cluster: osh @clustername [ ps ^ f 'p: (p.pid, p.commandline)' ] $