9 ms·

My rough workflow is: * Can I solve the problem using one-liners (grep, sed, awk, perl, sort, etc)? Or perhaps from within Vim? * Can I glue together one-line

by asicsp 4y ago

My rough workflow is: * Can I solve the problem using one-liners (grep, sed, awk, perl, sort, etc)? Or perhaps from within Vim? * Can I glue together one-liners with minimal control flow as a Bash script? * If not, go for Python --- Discussion for https://blog.jpalardy.com/posts/why-learn-awk/ https://blog.jpalardy.com/posts/why-learn-awk/ mentioned at the end of the article: https://news.ycombinator.com/item?id=22108680 https://news.ycombinator.com/item?id=22108680 (420 points | Jan 21, 2020 | 235 comments)

BiteCode_dev 4y ago
Same, but with 2 recent : - can I use ripgrep, fdfind, fzf and choose to do this? - can I ask chatgpt to do this ?
version_five 4y ago
If I have a text file of csv that I need to do something with, I'll usually start with a shell script and coreutils + sed and awk. If I need a script to generate some output (a regular use case that seems to come up for me is random numbers with some property), I tend to use python. I also use python if I need to do more complicated aggregations or something on tabular data that pandas is better at. Though it's fun to try with `join` and awk sometimes (parsing csv can get tricky). If I need to plot something I tend to use jupyter notebook but it's way more satisfying to use gnuplot, which I mention because it fits naturally into workflows that use shell tools like awk.
- masklinn 4y ago
  > If I have a text file of csv that I need to do something with, I'll usually start with a shell script and coreutils + sed and awk. You should install xsv.
  - version_five 4y ago
    I'm glad you mention that because I forgot something: I try and stick with only stuff I can find on most computers in order to minimize dependencies. That's a big advantage imo of learning how to use awk and tr for basic data analysis, for example. You can ssh into any server and work without installing anything.
    - goodoldneon 4y ago
      I agree 99%, but sometimes a tool is such a huge productivity boost that it’s worth installing. Fzf is one of those tools for me. If I had to deal with CSV files a lot then xsv would be worthwhile
    - cocodill 4y ago
      I don't like this approach. One is being peddled a bucket of old shit that might have been okay in the 80's for portability reasons. This portability is usually not necessary for my one time tasks anyway. Today there are really cool and small tools to do everything with comfort, and if you want some kind of portability you can still put static linked tools quickly on the given system. But of course that is a personal opinion and depends on the situation.
      - dev_tty01 4y ago
        > One is being peddled a bucket of old shit that might have been okay in the 80's for portability reasons. Like unix or linux? One person's "bucket of old shit" is another person's reliable swiss army knife that is always there when you need it.
        cocodill 4y ago
        Like people learning vi because it's part of POSIX and "is always there". For me the only thing that helped me was to learn :q!
        version_five 4y ago
        Really depends on your use case though. If you're always going to be working on your own machine, of course customize it to make yourself maximally productive. But if you're regularly going into new situations, it's good to know how to be effective with tools you can count on being there. My (admittedly very lame) anology: you're hiring a ninja assassin and have two candidates. One tells you about all the special swords and staffs and and smoke bombs he carries, the other says I just need my hands. Who do you hire?
  - tyingq 4y ago
    Sqlite is another option, and a fairly simple shell alias makes it nicer. https://sqlite.org/forum/info/ad9d1a8f3e9feb8bddf9be86749a2c4c683f1786d578faf4bc61fc9e5a2784ea https://sqlite.org/forum/info/ad9d1a8f3e9feb8bddf9be86749a2c...
  - chasil 4y ago
    Here is a script that I use, mainly composed with the Toybox version of awk, that will extract all of the WiFi passwords stored on an Android device. This is enormously portable, and does not require any new software installations for Android users who have root. Requiring xsv would reduce availability to a fraction of where it can be run now. #!/bin/sh find /data \ -name WifiConfigStore.xml \ -print0 | xargs -0 awk ' /"SSID/ { s = 1 } /PreShared/ { p = 1 } s || p { gsub(/[<][^>]+[>]/, "") sub(/^[&]quot;/, "") sub(/[&]quot;$/, "") gsub(/[&]quot;/, "\"") gsub(/[&]amp;/, "\\&") gsub(/[&]lt;/, "<") gsub(/[&]gt;/, ">") } s { s = 0; printf "%-32.32s ", $0 } p { p = 0; print } ' | sort -f Note that the -print0 null processing is not POSIX. This is a reasonable compromise of standards compliance, as it does not reduce the base of available users. I did try to do this first with arrarys, but awk segfaulted.
    - masklinn 4y ago
      XSV is a tool for exploring and manipulating X-separated value files (CSV, TSV, …), which is why I mentioned it in reply to a comment which talks about the exploration of CSV files, and furthermore specifically mentions that > parsing csv can get tricky
      - version_five 4y ago
        > tricky Yes I'd say awk is unsuitable for csv files that can contain text fields with arbitrary text, that includes commas, newlines, quotes etc. It gets hackier and becomes the wrong tool as the number of edge cases to handle increases.
        deleted 4y ago
        [deleted]
    - nousermane 4y ago
      That is quite nifty implementation of reverse HTML escaping. But in python that could be done with much less work: import html print(html.unescape(foo)) And the best part - you don't need to debug/update the (g)sub list every time you stumble upon new weird &whatever; too. And there are a lot of those out there: https://www.freeformatter.com/html-entities.html https://www.freeformatter.com/html-entities.html
- anthk 4y ago
  try xml2tsv git://bitreich.org/xml2tsv Then parsing it with AWK should be easy.
- asdff 4y ago
  Honestly for what you describe R might be quite comfortable. pandas was built to try and get R sensibilities into python, so you might as well drop the dependency and use the real deal all in one place.
gpvos 4y ago
This, but with Perl instead of Python. Perl really has a perfect flow from one-liners to small scripts.
- bsdooby 4y ago
  I use Tcl for such workflows...
  - tyingq 4y ago
    Tcl has pretty much no "one liner" capability though. No -e, nor other things like -n or -p.
    - bsdooby 4y ago
      That's true; but not really an issue, IMHO. If I am really keen for a one-liner, then (G)awk. [edited]
  - anthk 4y ago
    I would like Tcl if it didn't had upvar.
HopenHeyHi 4y ago
I suggest trying xsv as a first step: https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv
mananaysiempre 4y ago
I don’t think you’re right to put Awk into the one-liner category. It actually scales up remarkably well to a couple hundred lines or so, as long as the problem does not strain its anaemic data-structure capabilities. Compared to straight Python (i.e. not Numpy/pandas), it can also be surprisingly fast[1]. I experienced this personally on a trivial problem: take a TSV of (I,J) pairs, histogram I/J into a given number of bins. I can’t remember the exact figures now, but it went like this: on a five-gig file, pure Python is annoyingly slow, GNU awk is okayish but I still have to wait, mawk is fast enough that I don’t wait for it, C is of course still at least an order of magnitude faster but at that point it doesn’t matter. [1] https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-elegant-big-data-munging-language/ https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-...; note that the original author of mawk has made a release since then, at https://github.com/mikebrennan000/mawk-2 https://github.com/mikebrennan000/mawk-2, that doesn’t have the crashes I encountered with 64-bit builds of Dickey’s fork
- asicsp 4y ago
  Oh, I agree with you and the article that Awk is a wonderful scripting language. It is just that personally I use it mostly for one-liners (even wrote a book) and prefer Python for longer scripts.

by asicsp 4y ago

BiteCode_dev 4y ago

Same, but with 2 recent : - can I use ripgrep, fdfind, fzf and choose to do this? - can I ask chatgpt to do this ?

version_five 4y ago

If I have a text file of csv that I need to do something with, I'll usually start with a shell script and coreutils + sed and awk. If I need a script to generate some output (a regular use case that seems to come up for me is random numbers with some property), I tend to use python. I also use python if I need to do more complicated aggregations or something on tabular data that pandas is better at. Though it's fun to try with `join` and awk sometimes (parsing csv can get tricky). If I need to plot something I tend to use jupyter notebook but it's way more satisfying to use gnuplot, which I mention because it fits naturally into workflows that use shell tools like awk.

masklinn 4y ago
> If I have a text file of csv that I need to do something with, I'll usually start with a shell script and coreutils + sed and awk. You should install xsv.
- version_five 4y ago
  I'm glad you mention that because I forgot something: I try and stick with only stuff I can find on most computers in order to minimize dependencies. That's a big advantage imo of learning how to use awk and tr for basic data analysis, for example. You can ssh into any server and work without installing anything.
  - goodoldneon 4y ago
    I agree 99%, but sometimes a tool is such a huge productivity boost that it’s worth installing. Fzf is one of those tools for me. If I had to deal with CSV files a lot then xsv would be worthwhile
  - cocodill 4y ago
    I don't like this approach. One is being peddled a bucket of old shit that might have been okay in the 80's for portability reasons. This portability is usually not necessary for my one time tasks anyway. Today there are really cool and small tools to do everything with comfort, and if you want some kind of portability you can still put static linked tools quickly on the given system. But of course that is a personal opinion and depends on the situation.
    - dev_tty01 4y ago
      > One is being peddled a bucket of old shit that might have been okay in the 80's for portability reasons. Like unix or linux? One person's "bucket of old shit" is another person's reliable swiss army knife that is always there when you need it.
      - cocodill 4y ago
        Like people learning vi because it's part of POSIX and "is always there". For me the only thing that helped me was to learn :q!
        version_five 4y ago
        Really depends on your use case though. If you're always going to be working on your own machine, of course customize it to make yourself maximally productive. But if you're regularly going into new situations, it's good to know how to be effective with tools you can count on being there. My (admittedly very lame) anology: you're hiring a ninja assassin and have two candidates. One tells you about all the special swords and staffs and and smoke bombs he carries, the other says I just need my hands. Who do you hire?
- tyingq 4y ago
  Sqlite is another option, and a fairly simple shell alias makes it nicer. https://sqlite.org/forum/info/ad9d1a8f3e9feb8bddf9be86749a2c4c683f1786d578faf4bc61fc9e5a2784ea https://sqlite.org/forum/info/ad9d1a8f3e9feb8bddf9be86749a2c...
- chasil 4y ago
  Here is a script that I use, mainly composed with the Toybox version of awk, that will extract all of the WiFi passwords stored on an Android device. This is enormously portable, and does not require any new software installations for Android users who have root. Requiring xsv would reduce availability to a fraction of where it can be run now. #!/bin/sh find /data \ -name WifiConfigStore.xml \ -print0 | xargs -0 awk ' /"SSID/ { s = 1 } /PreShared/ { p = 1 } s || p { gsub(/[<][^>]+[>]/, "") sub(/^[&]quot;/, "") sub(/[&]quot;$/, "") gsub(/[&]quot;/, "\"") gsub(/[&]amp;/, "\\&") gsub(/[&]lt;/, "<") gsub(/[&]gt;/, ">") } s { s = 0; printf "%-32.32s ", $0 } p { p = 0; print } ' | sort -f Note that the -print0 null processing is not POSIX. This is a reasonable compromise of standards compliance, as it does not reduce the base of available users. I did try to do this first with arrarys, but awk segfaulted.
  - masklinn 4y ago
    XSV is a tool for exploring and manipulating X-separated value files (CSV, TSV, …), which is why I mentioned it in reply to a comment which talks about the exploration of CSV files, and furthermore specifically mentions that > parsing csv can get tricky
    - version_five 4y ago
      > tricky Yes I'd say awk is unsuitable for csv files that can contain text fields with arbitrary text, that includes commas, newlines, quotes etc. It gets hackier and becomes the wrong tool as the number of edge cases to handle increases.
      - deleted 4y ago
        [deleted]
  - nousermane 4y ago
    That is quite nifty implementation of reverse HTML escaping. But in python that could be done with much less work: import html print(html.unescape(foo)) And the best part - you don't need to debug/update the (g)sub list every time you stumble upon new weird &whatever; too. And there are a lot of those out there: https://www.freeformatter.com/html-entities.html https://www.freeformatter.com/html-entities.html
anthk 4y ago
try xml2tsv git://bitreich.org/xml2tsv Then parsing it with AWK should be easy.
asdff 4y ago
Honestly for what you describe R might be quite comfortable. pandas was built to try and get R sensibilities into python, so you might as well drop the dependency and use the real deal all in one place.

gpvos 4y ago

This, but with Perl instead of Python. Perl really has a perfect flow from one-liners to small scripts.

bsdooby 4y ago
I use Tcl for such workflows...
- tyingq 4y ago
  Tcl has pretty much no "one liner" capability though. No -e, nor other things like -n or -p.
  - bsdooby 4y ago
    That's true; but not really an issue, IMHO. If I am really keen for a one-liner, then (G)awk. [edited]
- anthk 4y ago
  I would like Tcl if it didn't had upvar.

HopenHeyHi 4y ago

I suggest trying xsv as a first step: https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv

mananaysiempre 4y ago

I don’t think you’re right to put Awk into the one-liner category. It actually scales up remarkably well to a couple hundred lines or so, as long as the problem does not strain its anaemic data-structure capabilities. Compared to straight Python (i.e. not Numpy/pandas), it can also be surprisingly fast[1]. I experienced this personally on a trivial problem: take a TSV of (I,J) pairs, histogram I/J into a given number of bins. I can’t remember the exact figures now, but it went like this: on a five-gig file, pure Python is annoyingly slow, GNU awk is okayish but I still have to wait, mawk is fast enough that I don’t wait for it, C is of course still at least an order of magnitude faster but at that point it doesn’t matter. [1] https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-elegant-big-data-munging-language/ https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-...; note that the original author of mawk has made a release since then, at https://github.com/mikebrennan000/mawk-2 https://github.com/mikebrennan000/mawk-2, that doesn’t have the crashes I encountered with 64-bit builds of Dickey’s fork

asicsp 4y ago
Oh, I agree with you and the article that Awk is a wonderful scripting language. It is just that personally I use it mostly for one-liners (even wrote a book) and prefer Python for longer scripts.