5 ms·
I was privileged to be one of the technical reviewers for this book. There's a fair bit of the original content (which is still great), but Kernighan's done a g
by benhoyt 3y ago
I was privileged to be one of the technical reviewers for this book. There's a fair bit of the original content (which is still great), but Kernighan's done a great job with some good restructuring and some significant updates, too. The early chapters are very hands-on, with something of a focus on "exploratory data processing", particularly with CSV files. Big data with AWK, you could say.
Gawk and awk will soon have a new "--csv" option that enables proper CSV input mode (parsing files with quoted and multiline fields per the CSV RFC). I'm really glad Arnold Robbins added a robust "--csv" implementation to Gawk, too, because that's really the most-heavily used version of AWK nowadays. I've already got CSV support in my own GoAWK implementation, and I'll be adding "--csv" to make it compatible.
I'm really glad this new updated version is coming out!
- Simon_O_Rourke 3y ago> Gawk and awk will soon have a new "--csv" option that enables proper CSV input mode Awesome!!!! Super excited to see this!
- calvinmorrison 3y agoIts a crying shame we never settled on a control character separated text format. There's a ascii control characters for record and field (unit) separators. A bit of user space support for that would have been great.
- galleywest200 3y agoIt is a shame. I have been using tab-separated sheets recently as it allows me to simply not care about almost any possible character in my strings...apart from tabs of course. But those are far less common than commas, and putting strings in quotes 100% of the time looks messy to me.
- calvinmorrison 3y agoWay less common would be using ascii 30 and ascii 31. ascii 29 and you can cram multiple datasets into one file
- bachmeier 3y agoAs I recall, you can tell Awk to use the control characters as record and field separators. Not helpful if you're getting your data from others, but if you're working by yourself, you have the option. I've come to use control characters as a default because it makes life so much easier.
- ufo 3y agoWhat do you recommend for viewing and editing such files?
- ac29 3y agoVisidata works with arbitrary separators. I just tried with a CSV separated with ␟ (ASCII unit separator) and it worked just fine.
- lolive 3y agoExcel too?
- leonim 3y agololive, VisiData has some Excel support. However, don't expect VisiData to be a full blown editor for Excel files. It can provide a view of the data in an Excel spreadsheet.
- dotancohen 3y agoI think GP means to ask if Excel can read files with ASCII 29,30,31 separators.
- smitty1e 3y agoIf you have a python installation available, openpyxl[1] is great both for converting to .csv and for packaging .csv outputs as .xlsx (which is really zipped .xml, anyway). [1] https://openpyxl.readthedocs.io/en/stable/index.html https://openpyxl.readthedocs.io/en/stable/index.html
- hermitcrab 3y agoSome discussion of that here: https://news.ycombinator.com/item?id=31220841 https://news.ycombinator.com/item?id=31220841 To be really useful as a format it would just need for text editors to: -display something distinct for the field separator (some editors do this) -treat the record separator character like a carriage return (not aware of any editors that do this)
- calvinmorrison 3y agoRight. The issue is the user space support at the end of the day.
- throw0101c 3y ago> To be really useful as a format it would just need for text editors to This made me think of WordPerfect's "reveal codes" functionality. :) (Word's "Reveal Formatting" is supposedly similar.)
- coldtea 3y ago>To be really useful as a format it would just need for text editors to: -display something distinct for the field separator Which would be trivial too.
- hermitcrab 3y agoThe programming might be straightforward. Trying to persuade the product owners to do it is a different matter.
- dotancohen 3y agoPull request?
- hermitcrab 3y agoAfter you. ;0)
- 3y ago
- JdeBP 3y agoMiller handles it. * https://miller.readthedocs.io/en/6.8.0/file-formats/#csvtsvasvusvetc https://miller.readthedocs.io/en/6.8.0/file-formats/#csvtsva... I have programs that handle it. * https://jdebp.uk/Softwares/nosh/guide/commands/console-flat-table-viewer.xml https://jdebp.uk/Softwares/nosh/guide/commands/console-flat-...
- PeterisP 3y agoTab-delimited "csv" formats are quite common (e.g. the CONLL format family for many natural language processing tasks) and also supported by common tools such as MS Excel for decades already.
- lolive 3y agoMost important comment I have ever read on HN ever !
- anyfactor 3y agoOur data product is delivered in CSV format. Even though I create user documentation mainly using csvkit, grep and sed, I would love to convert all those solutions to AWK. Sometimes AWK is more readable than sed and csvkit requires installation. It will be nice to have a awk cookbook for CSV. In terms of CSV maniupulation and querying there is only a limited number of operations and I think there is potential to standardize those operation using AWK.
- nmz 3y agoIt's nice that everyone is supporting this, I've written a portable awk module that takes control of the parsing and it is SLOW (and a little buggy). I'm a little bummed that nobody will use it but this is truly a step in the right direction. I guess for the people that are still using nawk, you can set up an AWK envvar so you can { awk -f $AWKU/ucsv.awk -f <(echo '{print NR, $1}') } https://github.com/Nomarian/Awk-Batteries/blob/master/Units/ucsv.awk https://github.com/Nomarian/Awk-Batteries/blob/master/Units/...
- tomcam 3y agoBen is not just any old technical reviewer. He wrote a version of AWK in go and has done a ton of other work in the AWK ecosystem.
- cauthon 3y agoThis is amazing, I may never use pandas again
- lost_tourist 3y agoWould you say the first few chapters are enough to get the 75-80% usefulness for mere mortals like me who will never try to master the full language? Or is the material fairly sprinkled throughout the whole tome?
- benhoyt 3y agoYes, definitely. The first three chapters would be more than enough for that: 1) An Awk Tutorial, 2) Awk in Action, and 3) Exploratory Data Analysis. For most people who just want to use AWK for one-liners on the command line, you can stop there. The rest of the chapters are about writing larger (still small! but not one-liner) programs in AWK to create reports, little languages, and experiment with algorithms.
- lost_tourist 3y agoThanks!
- ryenus 3y agoAwk is really great, for those knowing nvm [1], I used awk to make `nvm ls-remote` run more than 10 times faster [2] by replacing the related shell script with around 60 lines of awk script [3], and I was quite happy with the improvement. It's not really a one-liner, neither something big, but one can take that as an example regarding that awk is really not just for one-liners. Meanwhile having `--csv` support is really nice. I'd also like to see things like a builtin `length` function to be standard. [1]: https://github.com/nvm-sh/nvm/ https://github.com/nvm-sh/nvm/ [2]: https://github.com/nvm-sh/nvm/pull/2827/ https://github.com/nvm-sh/nvm/pull/2827/ [3]: https://github.com/nvm-sh/nvm/blob/9a769630d7/nvm.sh#L1703-L1765 https://github.com/nvm-sh/nvm/blob/9a769630d7/nvm.sh#L1703-L...
- benhoyt 3y agoBut length() is standard POSIX, no? Even length(array) has been approved by POSIX [1] but not yet included in the spec (they're very slow to update the spec for some reason). Both forms have been supported in onetrueawk, Gawk, mawk, and Busybox awk for a long time. [1] https://www.austingroupbugs.net/view.php?id=1566 https://www.austingroupbugs.net/view.php?id=1566
- ryenus 3y agoAh good to know, thank you! Meanwhile I found it, it was the awk from Debian 10 that lacked `length(array)`, see the related nvm pr [1]. [1]: https://github.com/nvm-sh/nvm/pull/2917/ https://github.com/nvm-sh/nvm/pull/2917/
- b3lm0nt 3y agoFantastic news. I’ve tried lots of new CLI tools but they always seem to fall between too little functionality (eg. xsv) and too much (VisiData). AWK is just right.