5 ms·

If I have a text file of csv that I need to do something with, I'll usually start with a shell script and coreutils + sed and awk. If I need a script to gener

If I have a text file of csv that I need to do something with, I'll usually start with a shell script and coreutils + sed and awk. If I need a script to generate some output (a regular use case that seems to come up for me is random numbers with some property), I tend to use python. I also use python if I need to do more complicated aggregations or something on tabular data that pandas is better at. Though it's fun to try with `join` and awk sometimes (parsing csv can get tricky). If I need to plot something I tend to use jupyter notebook but it's way more satisfying to use gnuplot, which I mention because it fits naturally into workflows that use shell tools like awk.

masklinn 4y ago
> If I have a text file of csv that I need to do something with, I'll usually start with a shell script and coreutils + sed and awk. You should install xsv.
- version_five 4y ago
  I'm glad you mention that because I forgot something: I try and stick with only stuff I can find on most computers in order to minimize dependencies. That's a big advantage imo of learning how to use awk and tr for basic data analysis, for example. You can ssh into any server and work without installing anything.
  - goodoldneon 4y ago
    I agree 99%, but sometimes a tool is such a huge productivity boost that it’s worth installing. Fzf is one of those tools for me. If I had to deal with CSV files a lot then xsv would be worthwhile
  - cocodill 4y ago
    I don't like this approach. One is being peddled a bucket of old shit that might have been okay in the 80's for portability reasons. This portability is usually not necessary for my one time tasks anyway. Today there are really cool and small tools to do everything with comfort, and if you want some kind of portability you can still put static linked tools quickly on the given system. But of course that is a personal opinion and depends on the situation.
    - dev_tty01 4y ago
      > One is being peddled a bucket of old shit that might have been okay in the 80's for portability reasons. Like unix or linux? One person's "bucket of old shit" is another person's reliable swiss army knife that is always there when you need it.
      - cocodill 4y ago
        Like people learning vi because it's part of POSIX and "is always there". For me the only thing that helped me was to learn :q!
        version_five 4y ago
        Really depends on your use case though. If you're always going to be working on your own machine, of course customize it to make yourself maximally productive. But if you're regularly going into new situations, it's good to know how to be effective with tools you can count on being there. My (admittedly very lame) anology: you're hiring a ninja assassin and have two candidates. One tells you about all the special swords and staffs and and smoke bombs he carries, the other says I just need my hands. Who do you hire?
        cocodill 4y ago
        I have a simple thought about this. If your work involves many machines, you usually have enough permissions to transfer your programs. I think that the benefits of a pleasant way of working outweigh the effort. And by the way, I would prefer the first ninja if the second one is handless.
  - masklinn 4y ago
    I guess if your passion is coercing broken tools into half-assign trivial jobs more power to you.
  - jcul 4y ago
    I'm surprised with some of the responses you've received on this. But I absolutely know what you are talking about. I often SSH into new machines, that I may not have root access on, that may not even have internet access, or be a distro with a package manager (e.g. some switch running a custom distribution). In those situations it is a huge advantage to know the tool, rather than try to do some gymnastics to get your tools onto the box or the data off the box. These "old" and "broken" tools are installed as defaults on all these systems for a reason.
- tyingq 4y ago
  Sqlite is another option, and a fairly simple shell alias makes it nicer. https://sqlite.org/forum/info/ad9d1a8f3e9feb8bddf9be86749a2c4c683f1786d578faf4bc61fc9e5a2784ea https://sqlite.org/forum/info/ad9d1a8f3e9feb8bddf9be86749a2c...
- chasil 4y ago
  Here is a script that I use, mainly composed with the Toybox version of awk, that will extract all of the WiFi passwords stored on an Android device. This is enormously portable, and does not require any new software installations for Android users who have root. Requiring xsv would reduce availability to a fraction of where it can be run now. #!/bin/sh find /data \ -name WifiConfigStore.xml \ -print0 | xargs -0 awk ' /"SSID/ { s = 1 } /PreShared/ { p = 1 } s || p { gsub(/[<][^>]+[>]/, "") sub(/^[&]quot;/, "") sub(/[&]quot;$/, "") gsub(/[&]quot;/, "\"") gsub(/[&]amp;/, "\\&") gsub(/[&]lt;/, "<") gsub(/[&]gt;/, ">") } s { s = 0; printf "%-32.32s ", $0 } p { p = 0; print } ' | sort -f Note that the -print0 null processing is not POSIX. This is a reasonable compromise of standards compliance, as it does not reduce the base of available users. I did try to do this first with arrarys, but awk segfaulted.
  - masklinn 4y ago
    XSV is a tool for exploring and manipulating X-separated value files (CSV, TSV, …), which is why I mentioned it in reply to a comment which talks about the exploration of CSV files, and furthermore specifically mentions that > parsing csv can get tricky
    - version_five 4y ago
      > tricky Yes I'd say awk is unsuitable for csv files that can contain text fields with arbitrary text, that includes commas, newlines, quotes etc. It gets hackier and becomes the wrong tool as the number of edge cases to handle increases.
      - deleted 4y ago
        [deleted]
  - nousermane 4y ago
    That is quite nifty implementation of reverse HTML escaping. But in python that could be done with much less work: import html print(html.unescape(foo)) And the best part - you don't need to debug/update the (g)sub list every time you stumble upon new weird &whatever; too. And there are a lot of those out there: https://www.freeformatter.com/html-entities.html https://www.freeformatter.com/html-entities.html
anthk 4y ago
try xml2tsv git://bitreich.org/xml2tsv Then parsing it with AWK should be easy.
asdff 4y ago
Honestly for what you describe R might be quite comfortable. pandas was built to try and get R sensibilities into python, so you might as well drop the dependency and use the real deal all in one place.

masklinn 4y ago

> If I have a text file of csv that I need to do something with, I'll usually start with a shell script and coreutils + sed and awk. You should install xsv.

version_five 4y ago
I'm glad you mention that because I forgot something: I try and stick with only stuff I can find on most computers in order to minimize dependencies. That's a big advantage imo of learning how to use awk and tr for basic data analysis, for example. You can ssh into any server and work without installing anything.
- goodoldneon 4y ago
  I agree 99%, but sometimes a tool is such a huge productivity boost that it’s worth installing. Fzf is one of those tools for me. If I had to deal with CSV files a lot then xsv would be worthwhile
- cocodill 4y ago
  I don't like this approach. One is being peddled a bucket of old shit that might have been okay in the 80's for portability reasons. This portability is usually not necessary for my one time tasks anyway. Today there are really cool and small tools to do everything with comfort, and if you want some kind of portability you can still put static linked tools quickly on the given system. But of course that is a personal opinion and depends on the situation.
  - dev_tty01 4y ago
    > One is being peddled a bucket of old shit that might have been okay in the 80's for portability reasons. Like unix or linux? One person's "bucket of old shit" is another person's reliable swiss army knife that is always there when you need it.
    - cocodill 4y ago
      Like people learning vi because it's part of POSIX and "is always there". For me the only thing that helped me was to learn :q!
      - version_five 4y ago
        Really depends on your use case though. If you're always going to be working on your own machine, of course customize it to make yourself maximally productive. But if you're regularly going into new situations, it's good to know how to be effective with tools you can count on being there. My (admittedly very lame) anology: you're hiring a ninja assassin and have two candidates. One tells you about all the special swords and staffs and and smoke bombs he carries, the other says I just need my hands. Who do you hire?
        cocodill 4y ago
        I have a simple thought about this. If your work involves many machines, you usually have enough permissions to transfer your programs. I think that the benefits of a pleasant way of working outweigh the effort. And by the way, I would prefer the first ninja if the second one is handless.
- masklinn 4y ago
  I guess if your passion is coercing broken tools into half-assign trivial jobs more power to you.
- jcul 4y ago
  I'm surprised with some of the responses you've received on this. But I absolutely know what you are talking about. I often SSH into new machines, that I may not have root access on, that may not even have internet access, or be a distro with a package manager (e.g. some switch running a custom distribution). In those situations it is a huge advantage to know the tool, rather than try to do some gymnastics to get your tools onto the box or the data off the box. These "old" and "broken" tools are installed as defaults on all these systems for a reason.
tyingq 4y ago
Sqlite is another option, and a fairly simple shell alias makes it nicer. https://sqlite.org/forum/info/ad9d1a8f3e9feb8bddf9be86749a2c4c683f1786d578faf4bc61fc9e5a2784ea https://sqlite.org/forum/info/ad9d1a8f3e9feb8bddf9be86749a2c...
chasil 4y ago
Here is a script that I use, mainly composed with the Toybox version of awk, that will extract all of the WiFi passwords stored on an Android device. This is enormously portable, and does not require any new software installations for Android users who have root. Requiring xsv would reduce availability to a fraction of where it can be run now. #!/bin/sh find /data \ -name WifiConfigStore.xml \ -print0 | xargs -0 awk ' /"SSID/ { s = 1 } /PreShared/ { p = 1 } s || p { gsub(/[<][^>]+[>]/, "") sub(/^[&]quot;/, "") sub(/[&]quot;$/, "") gsub(/[&]quot;/, "\"") gsub(/[&]amp;/, "\\&") gsub(/[&]lt;/, "<") gsub(/[&]gt;/, ">") } s { s = 0; printf "%-32.32s ", $0 } p { p = 0; print } ' | sort -f Note that the -print0 null processing is not POSIX. This is a reasonable compromise of standards compliance, as it does not reduce the base of available users. I did try to do this first with arrarys, but awk segfaulted.
- masklinn 4y ago
  XSV is a tool for exploring and manipulating X-separated value files (CSV, TSV, …), which is why I mentioned it in reply to a comment which talks about the exploration of CSV files, and furthermore specifically mentions that > parsing csv can get tricky
  - version_five 4y ago
    > tricky Yes I'd say awk is unsuitable for csv files that can contain text fields with arbitrary text, that includes commas, newlines, quotes etc. It gets hackier and becomes the wrong tool as the number of edge cases to handle increases.
    - deleted 4y ago
      [deleted]
- nousermane 4y ago
  That is quite nifty implementation of reverse HTML escaping. But in python that could be done with much less work: import html print(html.unescape(foo)) And the best part - you don't need to debug/update the (g)sub list every time you stumble upon new weird &whatever; too. And there are a lot of those out there: https://www.freeformatter.com/html-entities.html https://www.freeformatter.com/html-entities.html

anthk 4y ago

try xml2tsv git://bitreich.org/xml2tsv Then parsing it with AWK should be easy.

asdff 4y ago

Honestly for what you describe R might be quite comfortable. pandas was built to try and get R sensibilities into python, so you might as well drop the dependency and use the real deal all in one place.