5 ms·
The thing that prevents awk from being a major part of my daily routine is that it (amazingly) has poor CSV support. Consider the following: col1,col2,col3 1,
by jrumbut 5y ago
The thing that prevents awk from being a major part of my daily routine is that it (amazingly) has poor CSV support. Consider the following:
col1,col2,col3
1,2,3
4,"hello, \"world\"",6
"7 buckets",,9
To get the usual awk experience with this very common file format, exactly the type of thing you want to parse with awk, you first need to install gawk, then use a big FPAT regex that needs to be adjusted for any new CSV variant.
I would love to see awk with "CSV mode", where it intelligently handles formats like this if you just pass a flag. I think awk would do well to differentiate itself with excellent 2d dataset parsing functionality, but at least catchup up to the average scripting language would be great.
I'm half expecting someone to say "just pass -csv it does what you want" and if so I'll be very excited.
- nickcw 5y agoThere is an answer to CSV mode a bit further down the page https://news.ycombinator.com/item?id=28708145 https://news.ycombinator.com/item?id=28708145 ...but if your files are CSV, there is a CSV extension for gawk @include "csv" BEGIN { CSVMODE = 1 }
- jrumbut 5y agoWell there you go, for the sake of my pride at least it's an extension. It's funny searches for awk CSV seem to yield a bunch of SO questions where the answers are increasingly cumbersome regexes instead of this extension. Of course, you can't count of this extension being widely installed, but it's great for my own desktop.
- nmz 5y agothat's because the extension only works in gawk. its not portable anywhere else.
- sk5t 5y ago'miller' and 'xsv' are pretty good tools for wrangling CSV. (And regexp is kind of a terrible tool for it, too many edge cases.)
- jrumbut 5y agoYeah, I don't want to have to write a CSV library each time, that's what I'm trying to get at. I just end up using Python/Perl but I do have a soft spot for awk so it would be cool if good support was built-in.
- sk5t 5y agoWho's writing a library? Just use xsv or miller to extract the bits you want from the CSV, change the delimiter or escapes to something more convenient, etc., then feed that to awk or other CSV-unaware text processors.
- jrumbut 5y agoI was agreeing with your point about regexes, that it's good to avoid trying to deal with all the corner cases yourself when you're just trying to write a small script.
- sk5t 5y agoAh, understood! CSV is funny, it seems like a more trivial thing than it really is, and its human readability sort of invites broken approaches in a way that something like Parquet would not. XML is somewhere in the middle--I've seen some horrible abuses of CDATA sections way back when--but at least there are accepted ways to prove what's invalid.
- nmz 5y agoYou can just use https://github.com/Nomarian/Awk-Batteries/blob/master/Units/ucsv.awk https://github.com/Nomarian/Awk-Batteries/blob/master/Units/... and use as so awk -f ./ucsv.awk -e '{print $5}' Also this > 4,"hello, \"world\"",6 Is incorrect per https://tools.ietf.org/html/rfc4180 https://tools.ietf.org/html/rfc4180 so you should just fix it with a sed -i 's/\\"/""/g' and then just parse as normal. https://github.com/Nomarian/Awk-Batteries/wiki/Formats https://github.com/Nomarian/Awk-Batteries/wiki/Formats
- dbro 5y agoThere is a small program I wrote called csvquote[1] that can be used to sanitize input to awk so it can rely on delimiter characters (commas) to always mean delimiters. The results from awk then get piped through the same program at the end to restore the commas inside the field values. In principle: cat textfile.csv | csvquote | awk -f myprogram.awk | csvquote -u > output.csv Also works for other text processing tools like cut, sed, sort, etc. [1] https://github.com/dbro/csvquote https://github.com/dbro/csvquote