18 ms·
Why Learn Awk? (2016)
- sremani 7y agoHonorary mention of Taco Bell Programming. (fits this genre). http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra... Someone ought to write - Zen and the art of Unix tools usage.
- dhruvkar 7y ago>> suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? I don't know enough about the 'real way' or the 'taco bell way', but interested to know --- is this doable the way Ted describes in the article via xargs and wget?
- msla 7y agoSimple, Composable Pieces is practically the whole ethos of both Unix and Functional Programmers, and they both converged on the same basic flow model: Pipelines with minimal side-effects. This naturally leads to a sequential concurrency, which seems to be easy for humans to reason about: Data flows through the pipeline, different parts of the pipeline can all be active at once, and nobody loses track. It doesn't solve absolutely every possible problem, but the right group of pieces (utility programs, functions) will solve a surprising number of them without much trouble.
- PaulDavisThe1st 7y agoWorks until time plays a role in the computation you're doing.
- peterwwillis 7y ago(warning, mandatory HN contrarian comment) "This is the opposite of a trend of nonsense called DevOps, where system administrators start writing unit tests and other things to help the developers warm up to them - Taco Bell Programming is about developers knowing enough about Ops (and Unix in general) so that they don't overthink things, and arrive at simple, scalable solutions" It's not possible for developers to know enough about Ops, just as it's not possible for Ops to know enough about development, because they are different jobs. Moreover, devs are doomed to create terrible solutions because of their job, and Ops are doomed to create kludgy hacks for those terrible solutions because of their job. DevOps is just an attempt to get them to talk to each other frequently so that horrible shit doesn't happen as frequently. Also, the real Taco Bell programming is actually to use only wget, no xargs. It takes a whole lot of basically every option Wget has, and a very reliable machine with a lot of RAM, but you can crawl millions of pages with just that tool. xargs and find make it worse because you don't get the implicit caching features of the one giant threaded process, so you waste tons of time and disk space re-getting the same pages, re-looking up the same hostnames, etc. (And that's Ops knowledge...) The Zen of Unix is to try to move towards not using the computer at all. One-liners are part of that path, but so is minimizing the one-liner. http://www.catb.org/~esr/writings/unix-koans/ten-thousand.html http://www.catb.org/~esr/writings/unix-koans/ten-thousand.ht...
- gnulinux 7y agoI don't understand this devs don't understand ops nonsense. > so you waste tons of time and disk space re-getting the same pages, re-looking up the same hostnames, etc. (And that's Ops knowledge...) If it's my job to write a web scraper, it's absolutely my job to think about/solve this problem. Is this a new trend thing?
- deleted 7y ago[deleted]
- peterwwillis 7y agoThe difference between dev & Ops is an Italian grandma vs a restaurant chef. It's different experience that gives you different knowledge and a different skillset.
- YarickR2 7y agoBecause you often don't need anything else to deal with a surprisingly large class of tasks in IT.
- deleted 7y ago[deleted]
- dhruvkar 7y agoNeeds a '(2016)' at the end of the title.
- LinuxBender 7y ago[updated] ty.
- LinuxBender 7y agoHere are some clever one-liners for awk [1] Please be sure to add your own. [1] - https://www.commandlinefu.com/commands/matching/awk/YXdr/sort-by-votes https://www.commandlinefu.com/commands/matching/awk/YXdr/sor...
- geogra4 7y agoreminds me of the old Knuth vs McIlroy word count battle
- asicsp 7y agoI have a repo dedicated for some of the cli text processing tools like grep/sed/awk/perl/sort/etc. Here's my one-liner collection for awk [1] [1] https://github.com/learnbyexample/Command-line-text-processing/blob/master/gnu_awk.md https://github.com/learnbyexample/Command-line-text-processi...
- LinuxBender 7y agoThat is a very well written set of examples. Clear and concise. Thankyou!
- canada_dry 7y agoWow... a great resource! It's very well written and consistent.
- Afton 7y ago(2016)
- jerf 7y agoBecause for some bizarre reason, "cut" doesn't ship with any decent column selection logic that is the equivalent of awk's $1, $2, etc., even in 2020. That's like 90% of my use of awk right there. I don't know of any easier equivalent of "awk '{ print $2 }'" for what it does. Posted partially so the Internet Correction Squad can froth at the mouth and set me straight, because I'd love to be showed to be wrong here.
- pizza234 7y ago> I don't know of any easier equivalent of "awk '{ print $2 }'" for what it does. I'm not sure if you refer spefically to cut, but Perl has something similar and approximaly terse: > echo 'a b c' | perl -lane 'print $F[1]' Also, Perl can slice arrays, which is something that I really miss in Awk.
- jerf 7y agoThat is closest, yes. I'd say clearly a couple more things to remember, but if I can get it into my fingers will be just as fluid. Awk's invocation has its own issues around needing to escape the things the shell thinks it owns, too, not that it's at all unique about that.
- 0xff00ffee 7y agoPERL is bloatware by comparison and less likely to be installed on distros than AWK. (e.g, embedded or slim distros. that's why you rarely see nonstandard /bin execs in shell scripts).
- sigjuice 7y agoThe awk on an embedded system is most likely a non-mainstream awk implementation with fewer or different features.
- 0xff00ffee 7y agoProof? Or are you just guessing?
- rectang 7y agoAwk, like shell, is a fraught programming environment, full of silent failure and hidden gotchas. Even for one-liners. I use shell because I have to, not because I like it. I dread maintaining shell scripts which have a bunch of awk and sed in them. The Unix ideal of small single-purpose tools and text processing is separable from these old warhorses.
- rb808 7y agoscripts are the antithesis of modern software development. No Unit testing, No CI, no monitoring, no consistency, often no source control. Appeals to me as a hacker but only my scripts are intelligible - which is a bad sign. :)
- sansnomme 7y agoPowerShell begs to differ.
- umvi 7y agoI'm not actually convinced unit testing is all that valuable, unless the unit under test has a very clear input -> output transformation (like algorithms, string utilities, etc). If it doesn't (and most units don't), unit tests just encumber you.
- Twisol 7y agoThe value of unit tests when creating software is perhaps debatable, but I find their greatest value to be in maintaining software. If you lock in your expectations of a software component, when it's time to make changes (due to shifting requirements or what not), you can add tests capturing your intent and make sure that you aren't breaking existing expectations. Or at least, that you know which expectations your change will break.
- Eikon 7y agoI’m really glad to read that. I never understood the whole religion around unit tests. Integration tests are often far easier to write and far more valuable. Like you said, unit tests are really nice when testing for a known, expected output. Unit tests that are effectively testing mocks and crazy stubs because your method has side effects? Not for me.
- hawski 7y agoI love awk, but I'm still waiting for the structural regex awk: http://doc.cat-v.org/bell_labs/structural_regexps/se.pdf http://doc.cat-v.org/bell_labs/structural_regexps/se.pdf
- linsomniac 7y agoI just learned about SE a month or so ago, and it is indeed pretty awesome. I tried out the "vis" editor, using SE to create multiple cursors that I then manipulated with vi-like commands. That was a pretty sweet use case.
- rhombocombus 7y agoIt is fast, robust, and frequently far more performant than a lot of modern tools that can be overkill for most data manipulation. I use it all the time in our ETL processes and it always works as advertised.
- shpeedy 7y agoPerl is much faster[0], with much more features, with bunch of ready to use libraries, with package manager (CPAN), and similar syntax to awk. Why you use awk? [0]: http://rc3.org/2014/08/28/surprisingly-perl-outperforms-sed-and-awk/ http://rc3.org/2014/08/28/surprisingly-perl-outperforms-sed-...
- Scarbutt 7y agoBecause you can learn awk in 1/10 of a time it takes to learn perl.
- rhombocombus 7y agoI use it for a couple reasons: one, it is installed as a base app on almost every single *nix implementation on the planet, so you can count on having it even on ancient or restrictive environments (which I work in frequently); Two, awk is frequently fast enough for most needs, and generally far faster than a number of off the shelf "modern" tools. The first reason is the one that generally leads me to its use, its ubiquity and power make it a compelling tool.
- macintux 7y agoPerl was my second programming love, but awk is much shorter and easier to remember for the simple cases where I need it. Remembering which Perl command-line arguments simulate awk’s line-by-line processing is harder than just remembering awk.
- mekazu 7y agoI use Perl similarly to awk if I need to use regex rather than white space delimited fields. I think if you know Perl really well and can remember the command line arguments - particularly -E, -n, -I and -p - then it’s a good swap in substitute for grep, sed, awk, cut, bash, etc when whatever 5 min task you’re working on gets a tiny bit more complex. Similarly a decent version perl 5 seems to be installed everywhere by default. I’m curious to know if anyone would say the same about python or any other programs? I’m not particularly strong in short python scripting.
- jmercouris 7y agoWhy not just use Emacs for simple scripts and text manipulation? Way easier, lots of one liners as well, just as expressive.
- Chris2048 7y agoHmm. I'm sure this question will induce a flamewar of practical "#NeverAwk"-ers fighting toolbelt bloat, versus tech-hoarding AWK apologists arguing against throwing something out given if fills <niche>. Here's the thing: these arguments all too commonly focus on subjective notions of "simplicity", and toy examples divorced from actual common practise, and or solid comparable benchmarks. Show me a range of practical examples, for each competing env (awk, sed, perl, python, ruby, maybe bash). Include: - time it takes to teach a total novice (the time it takes to learn whatever is needed that example, not the entire language) - how easy it is to recall said knowledge at a later date - how fast example is multiplied by how much you are likely to use it = actual time saved in terms of execution. for small, fast examples the difference is irrelevant, a 10x speedup that is 0.1s vs 0.01s is meaningless. - how extendable an example is. Hence the original example should include a series of extensions to the original task, to demonstrate how flexible / composable they are: e.g. task 1) count lines in a file; task 2) count lines in a file, then add 42 to it; I suspect awk falls behind in practicality vs perl (which can do simple one-liners, but also more complicated constructs), but perhaps has a hidden virtue wrt speed in more expensive tasks, ala https://news.ycombinator.com/item?id=17135841 https://news.ycombinator.com/item?id=17135841 or https://news.ycombinator.com/item?id=20293579 https://news.ycombinator.com/item?id=20293579
- arminiusreturns 7y agoI like to tell the story of when I was doing some genetics data wrangling and spent three days writing some perl code and I kept failing, then sent one of the researchers an email and he suggested an awk method and I turned 3 pages of perl into an awk one liner that works just fine. Now, its probably because I don't know perl very well, but as an ops type who doesnt have the classical dev/cs education, tools like it are my goto.
- bonoboTP 7y agoI find Python list/dict comprehensions, zip, range, enumerate etc. and the itertools module very good for such things. And you have immediate access to so many useful modules (csv, json, xml) and can easily extract code fragments into functions. You can also execute shell-like commands with subprocess.check_output() without ever worrying again about escaping strings or accidentally splitting them at spaces or whatever. Clever one liners are difficult to comprehend. It's better to break them up to a few variable assignments with descriptive, long names without abbreviations.
- samatman 7y ago> Imagine programming without regular expressions. I live in this future and it's beautiful. Steps to programming without regular expressions: 1) find a PEG library for your language of choice There is no step 2.
- iagovar 7y agoSorry for asking such dub question, but what's a PEG library?
- tejtm 7y agoParsing expression grammar; It is a recursive decent parser with the tiny tweak that productions are ordered (not a set) and short circuit. A 90's language you did not have to imagine that saved you from regex in this way was[is] REBOL with it built in `parse` dsl. examples here http://www.rebol.org/search.r?find=parse&form=yes http://www.rebol.org/search.r?find=parse&form=yes [0]https://en.wikipedia.org/wiki/Parsing_expression_grammar https://en.wikipedia.org/wiki/Parsing_expression_grammar [1]http://www.rebol.com/ http://www.rebol.com/
- chrisjc 7y agoNo idea... had to look it up. Parsing Expression Grammar. https://en.wikipedia.org/wiki/Parsing_expression_grammar https://en.wikipedia.org/wiki/Parsing_expression_grammar
- linsomniac 7y agoTL;DR: PEG looks to me like simplified lex+yacc For those not familiar with PEG, like me, here: https://en.wikipedia.org/wiki/Parsing_expression_grammar https://en.wikipedia.org/wiki/Parsing_expression_grammar I found that fairly abstract, a Python example provided more concrete examples: https://github.com/erikrose/parsimonious https://github.com/erikrose/parsimonious
- nickpsecurity 7y agoQuite a few alternatives to RE's I put in this answer: https://softwareengineering.stackexchange.com/questions/194975/readable-regular-expressions-without-losing-their-power/195013#195013 https://softwareengineering.stackexchange.com/questions/1949...
- loriverkutya 7y ago"Imagine programming without regular expressions." Best of all worlds. I wish there would be a way to get back the time I spent with debugging edge cases of different regex implementations on various OS's.
- shpeedy 7y agoSkip learning of sed and awk and jump straight to perl instead. $ perl --help ... -F/pattern/ split() pattern for -a switch (//'s are optional) -l[octal] enable line ending processing, specifies line terminator -a autosplit mode with -n or -p (splits $_ into @F) -n assume "while (<>) { ... }" loop around program -p assume loop like -n but print line also, like sed -e program one line of program (several -e's allowed, omit programfile) Example. List file name and size: ls -l | perl -ae 'print "@F[8..$#F], $F[4]"'
- just_myles 7y agoBecause syntatically as a language/tool it is super easy to remember. Writing one liners with awk feels more intuitive to me. Awk example: ls -l | awk '{print $9, $5}' or ls -lh | awk '{print $9, $5}' Seems a whole lot simpler. To me. I find if you have to write exhaustive shell scripts then maybe you can look for something more verbose like Perl, I guess.
- shpeedy 7y agoYep, but you have bug in your awk one-liner.
- wahern 7y agoIf you mean the lack of quotations, then the behavior is well-defined and is presumably what was intended. Per POSIX, > The print statement shall write the value of each expression argument onto the indicated output stream separated by the current output field separator (see variable OFS above), and terminated by the output record separator (see variable ORS above). The default value for OFS is <space> and for ORS, <newline>.
- demiol 7y ago> If you mean the lack of quotations, No, lack of commas in output and broken filenames with spaces.
- taxtropel 7y agoBecause it's awksome!
- mason55 7y agoAnother good resource is "Why you should learn just a little Awk: An Awk tutorial by Example"[0] [0] https://gregable.com/2010/09/why-you-should-know-just-little-awk.html https://gregable.com/2010/09/why-you-should-know-just-little...
- rwnspace 7y agoInstead, learn fex: https://www.mankier.com/1/fex https://www.mankier.com/1/fex
- geocrasher 7y agoMy favorite Awk reference is this: https://ferd.ca/awk-in-20-minutes.html https://ferd.ca/awk-in-20-minutes.html Also, a handy trick is to combine awk and cut. For example I had a log line that had a variable amount of columns just in one field, but immediately after the field was a comma. I cut based on the comma: cut -d, -f1,2 and then awk'd the last column: cut -d, -f1,2 | awk '{ print $2" "$5" "$NF }' So, sometimes awk and cut can help each other.
- ubadair 7y agoI've been procrastinating learning awk for a while now. Thanks for this, I read it and it's just what I needed.
- berkes 7y agoLast week I threw out AWK and replaced it with Ruby (Could've been Python, Perl or PHP even). Because AWK is not suited for CSV. Please prove me wrong! I had to parse 9million lines. Some of which contain "quoted records", others, same column, are unquoted. Some contain comma's, in the fields, most don't. CSV is like that: more like a guideline than actual sense. Two hours of googling and hacking later, I gave up and rewrote the importer in Ruby, in under 5 minutes. Lesson learned: I'll stay clear of AWK, when I know a oneliner of Ruby (or Python) can solve it just as well. Because I know for certain the latter can deal with all the edgecases that will certainly pop up.
- ende 7y agoIf you’re doing a lot of csv ops on the cli, you might like https://csvkit.readthedocs.io/en/latest/ https://csvkit.readthedocs.io/en/latest/ (It’s Python, and you can use it as a library as well)
- mandelbrotwurst 7y agoI'm just starting to learn awk, would you mind sharing some of the issues / edge cases that you ran into that you found it didn't handle well? Was it maybe just that you found it tricky to write the regular expressions?
- F-0X 7y ago> I had to parse 9million lines. Awk would chew through that no problem. > Some of which contain "quoted records", others, same column, are unquoted. In which case, there is the FPAT variable which can be used to define what a field is. FPAT="\"[^\"]\"|[^,]", which means "stuff between quotes, or things that are not commas", would probably have worked for you. (EDIT: Looks like formatting has gotten hold of my FPAT and I don't know how to stop it... hopefully it is still clear where asterisks should be) > Some contain comma's, in the fields, most don't. CSV is like that: more like a guideline than actual sense. Well, I would say that's absolutely false. You can't just put the delimiter wherever you fancy and call it a well-formed file. Quoting exists for the unfortunate cases your data includes the delimiting character (ideally the author would have the sense to use a more suitable character, like a tab). This is just a retort to prevent your post from dissuading readers from awk, which is a fantastic tool. If you actually sit for half and hour and learn it rather than google to cobble together code that works, it is wonderful. I also don't think it is valid to base your judgement of a tool on what was apparently garbage data.
- eska 7y agoI gave awk a sincere attempt, but I have to say that it wasn't worth it. As soon as one tries to write anything bigger than a one liner the language shows its warts.. I found myself writing lots of helper routines for things that should be part of the base language/library, e.g. sets with existence check. I also had to work around portability issues, because awk is not awk, unlike this post claims. E.g. matching a part of a line is different among different awks. And some language decisions are just asinine, e.g. how function parameters and variable scope works, fall through by default although you almost never want that, etc.. But hey, you have socket support! Sounds to me like things have developed in the wrong direction. And of course no one on your team will know awk. I found the idea of rule-based programming interesting, but the way it interacts with delimiters and sections (switching rules when encountering certain lines) doesn't work well in practice. I also found the performance to be very disappoinging when I compared it to C and python equivalents.
- 0xff00ffee 7y agoYou realize AWK is about 1/100th the size of Python, right? That's like comparing a Leatherman multi-tool to a Craftsman 2000 piece tool set that weighs 1,000 lbs. This matters significantly when addressing compatibility and when building distros that are space constrained. Awk is there for a reason: to be small. That's why the O'Reilly press book is called "Sed & Awk", because they were orignally written to work together in the early days of unix dating back to the late 70's. Sed (1974) & Awk (1977) are in the DNA of unix, Python is something totally different.
- eska 7y agoFirst of all I'm not a distro maintainer. I also doubt that people would use awk for seriously space constrained environments. And distros ship both awk and python anyway. And again, I don't understand why they'd support networking but not basic data types/functions. The only reason I could've seen to use awk was to throw code together more quickly in a DSL. However this is much less the case than I had hoped. For the one liners there are usually specialized tools like fex that are easier to use and faster (for batch processing). When I compared my C/python/awk programs the difference was msec/sec/minutes. As soon as I use such a program repeatedly it starts to hurt my productivity. And the development time is not orders of magnitude slower in non-awk languages.
- bloopernova 7y agoI use awk because there's an almost 100% chance that it's going to be installed on any unix system I can ssh into. I use awk because I like to visually refine my output incrementally. By combining awk with multiple other basic unix commands and pipes, I can get the data that I want out of the data I have. I'm not writing unit tests or perfect code, I'm using rough tools to do a quick one-off job. For instance, "mail server x is getting '81126 delayed delivery' from google messages in the logs, find out who is sending those messages". # get all the lines with the 81126 message. Get the queue IDs, exclude duplicates, save them in a file. cat maillog.txt | grep 81126 | awk '{print $6}' | sort | uniq | cut -d':' -f1 > queue-ids.txt # Grep for entries in that file, get the from addresses, exclude duplicates. cat maillog.txt | grep -F -f queue-ids.txt | grep 'from=<' | awk '{print $7}' | cut -d'<' -f2 | cut -d'>' -f1 | sort | uniq Each of those 2 one-liners was built up pipe-by-pipe, looking at the output, finding what I needed. It's not pretty, it's not elegant, but it works. I'm sure there's a million ways that a thousand different languages could do this more elegantly, but it's what I know, and it works for me.
- darau1 7y ago> it's not elegant I completely disagree.
- samstave 7y agoEloquently put!
- bloopernova 7y agoThank you, I too often talk down what I do.
- rhombocombus 7y agoIts ubiquity and performance open up all kinds of sophisticated data processing on a huge variety of *nix implementations. Whether it's one liners or giant data scrubs, awk is a tool that you can almost always count on having access to, even in the most restrictive or arcane environments.
- 7y ago
- 0xff00ffee 7y agoWhen writing a shell script the robustness/reliabilty can be inferred from it's scope: 1. Uses shell-only commands (echo, for) - most robust; but things like basename/dirname and regex's vary by shell (sh, bash, zsh, ksh) 2. Uses /bin - might run into missing a binary but not likely, still robust and allows a richer set of tools (e.g., uname, chmod and admin-ish things live in /bin) 3. Uses /usr/bin - runs risk if missing packages, likely not very robust (packages drop things in here, like gzip, yacc, gcc) 4. Uses /usr/local/bin or /opt/local/bin - definitely requires package installs, least robust
- technofiend 7y agoParenthetically, since there are a bunch of UNIX greybeards on HN: if anyone has artwork of the AWK t-shirt I will happily pay any reasonable price. The shirt has a parachute-wearing bird about to jump from an airplane and is captioned with AWK's most famous error message: awk: bailing out near line 1. Contact information is hn handle @ yahoo.com.
- Communitivity 7y agoAwk is a command I turn to time and again. For me it's the single most valuable command for enabling the piped single-purpose pattern. As an example, if I want a sorted list of all open files under the home directories on CentOs I can do this: lsof | awk '{ print $10 }' | grep ^/home/ | sort | uniq
- pge 7y agoyou can save one command by using sort -u which eliminates dupes in the sort, removing the need for uniq
- kbr2000 7y agoTnx, that one I didn't know... You can drop another command as such: lsof |awk '($10 ~ /^\/home\//) {print $10}' |sort -u
- kmstout 7y agoDon' stop there. You've solved a real problem in your life, and you might want that information another day. Make a tiny script that encapsulates it. Generalize it a tiny bit, and give it a memorable name (perhaps lsof-tree). That done, you can stop worrying about the mechanics of the solution and build on it. #! /bin/bash # lsof-tree: list open files in a given directory tree (default /home) NAME=9 # set to 10 for CentOS BASE="${1:-/home}" lsof | awk -v NAME=$NAME '{print $NAME}' | grep "^$BASE" | sort -u
- jabl 7y agoYMMV, but I find it easier and faster to know the basic utilities and how to compose them with pipes than remembering the name of a zillion such scripts. I guess it depends on how often you need that particular pipeline. Every day? Sure, make a script. Every few months? Nah, I won't remember it anyway,or probably I remember that I've made a script like that but then I have to start searching my bin directory in the end using more time than just writing the pipeline in the first place.
- 3xblah 7y agoThe author forgot one: C-like syntax I believe this was intentional by the authors. In the early days of UNIX, I think more of its users knew C. Today, it is probably a much smaller number. However, learning AWK today, IMO, can help someone who also intends to learn C.
- shmerl 7y agoSo why for example Awk instead of let's say Perl? The argument of "it's likely to be installed" isn't very compelling.
- dredmorbius 7y agoIf you're versed in both, it comes down to taste, though there really are cases where you'll have awk (usually via busybox) but not Perl. OpenWRT comes to mind (just verified it's not present by default, though yes, packages are available). For a huge number of simple tasks, awk is available and sufficient. It's largely a subset of Perl, so yes, there's some skills overlap, but there are times where knowing awk is the right tool and the available tool will pay off.
- shmerl 7y agoI like Perl regexes more though, especially aliases. Using \d is a lot neater than [0-9] or [[:digit:]].
- dredmorbius 7y agoI hear Perl may support those ;-) (Use the right tool for the job.)
- arendtio 7y agoDoes someone know how I have to invoke awk to make sure that the awk code I wrote is POSIX compliant?
- znpy 7y agodo you know that feeling when a problem is too much for the shell but too little for C? that's where Perl was supposed to fit. in everyday life however, many small problems are a bit too much for the shell but too little for Perl/Python/whatever. awk fits very nice in there.
- jibbit 7y agoThere used to be a website shared on HN a lot, which was a table of: Task, SED, Awk. It was really useful but I haven’t seen it for years.
- doctor_eval 7y agoIt would be so great if awk had a csv mode. For whatever reason (Excel), CSV seems to be the default text format for field oriented data. Maybe I’m dumb but I’ve never come up with a separator regex that is quite right.
- gpanders 7y agoFor _actually_ comma separated values, just use awk -F , '...' Is this not what you mean? Edit: This comment might be helpful to you: https://news.ycombinator.com/item?id=22110036 https://news.ycombinator.com/item?id=22110036
- doctor_eval 7y agoThe problem is that the field themselves can contain quotes ('"'), which escapes the comma. So the standard FS=/-f doesn't work properly. It looks like FPAT from your linked article is for gawk. Gawk is great, but it's not everywhere. Still - it's good to know. Thanks!
- dredmorbius 7y agoThe simple case is: FS = "," or: awk -F ',' <program> If you're working with CSV data that has quoted strings with embedded commas, FPAT is your friend: FPAT = "([^,]+)|(\"[^\"]+\")" See: https://www.gnu.org/software/gawk/manual/gawk.html#Splitting-By-Content https://www.gnu.org/software/gawk/manual/gawk.html#Splitting...
- doctor_eval 7y agojust want to say thanks for this, I haven't had to deal with CSV for years and now only a week after you posted this I needed it. brew install gawk good to go :)
- jackcosgrove 7y agoAfter a decade of writing application software in C family languages, I am now working on a big devops effort in a Linux environment. A lot of the syntax is janky but it's pretty amazing what you can accomplish with a shell and the Linux command line tools.
- justincredible 7y agoDo people refer to maps as hashes?
- ineedasername 7y agoIn the early aughts, fresh out of college, one of my first projects was to take semi-structured text from database blobs and convert it to XML. I quickly realized it was not a task to be fully automated, too many edge cases that really required human judgement because the it was only very loosely semi-structured. I turned to sed, awk, and pico. Sed & awk did the 99%, and dumped into pico when it didn't know what to do and I would resolve the issue. Doing it semi-supervised in this way was 100 times faster than doing it all manually, and 10 times faster than full automation, and probably more accurate. But it was the ability to string together these types of command line tools that made it possible.
- dredmorbius 7y agoI've been parsing some documents converted from PDF (using the Poppler library's "pdftotext" command with the "--layout" option). I found that reading these -- sort-of half-assed structured data, but with page-chunked artefacts and idiosyncrasies -- was difficult on a line-by-line basis, and thought idly "this would be a lot easier if I could process by page instead". Text was laid out in columns, and the amount of indenting (the whitespace between columns) was significant. So preserving this somehow would be Very Useful. Suddenly those pesky '^L' formfeeds were an asset, not a liability. Let's treat the formfeed ("\f") as a record delimiter, and the newline ("\n") as a field delimiter. We can parse out the actual columns based on witespace, for each line: BEGIN { RS="\f"; FS="\n" } { pageno = NR lines = NF for( line=1; line<=lines; line++ ) { ncols = split( $line, columns, " {2,}", gaps ) } } This gives me: - The running tally of pages. - Each line of the page as an individual record. - Via the split() function, an array of columns separated by two or more spaces, which are saved as an array of gaps so I have the whitespace to play with. Edge cases and fiddling ensue, but that's the essential bit of the code there. Since the lines are an array, I can roll back and forth through the page (basically being able to read forward and backwards through the text record), testing values, finding out where column boundaries are, etc., and then output a page's worth of content, transposing to a single-column format, with appropriate whitespacing, when done. In testing and debugging the output (working off of 20+ documents of 100s to ~1,000 pages), a lot of test cases, scaffolding, diagnostics, etc., have been created and removed to make sure the Right Things are happening. Easy with awk.
- dredmorbius 7y agoAnd to be clear: After realising this ... and knowing what to look for ... I found this documented in the GNU Awk User's guide: https://www.gnu.org/software/gawk/manual/html_node/Multiple-Line.html https://www.gnu.org/software/gawk/manual/html_node/Multiple-...
- AzzieElbab 7y agoI have used awk for decades, but I simply stoped and dropped the habit of using it along with sed and perl. Nowadays I would rather write a program than a script just to avoid memorizing all these tricks and hacks and glitches