19 ms·
Learn just a little Awk (2010)
- MaDeuce 8y agoI was first exposed to Awk when I started work at Bell Labs in the late 80s. Until then, I'd been using either Lisp or C exclusively and was really blown away by how simple some things were in Awk. I used it with impunity to munge all sorts of data for input into fault prediction tools I was working on. Speed was never an issue for me, so I never explored the potential improvements offered by 'awkcc'. Although perl was becoming the new hotness at that time, Awk remained my goto tool for many years. If you are interested in learning Awk, I highly recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger. It's about the same size as the original "The C Programming Language" and is equally well-written. Previously on HN: https://news.ycombinator.com/item?id=13451454 https://news.ycombinator.com/item?id=13451454
- teachrdan 8y ago> Aho, Kernighan, and Weinberger That must be where the name comes from, right?
- dbolgheroni 8y agoYes.
- int0x80 8y agoha! TIL
- EvilTerran 8y agoNote that's the same Kernighan of "Kernighan & Ritchie" fame. Very influential feller.
- chrisfinazzo 8y agoIt's fitting. Their names, and it's also awkward AF to write. The only better example is probably Brainfuck.
- electricslpnsld 8y agoAho taught the programming languages course at my Uni, and he loved to tell stories about Bell Labs, and this Brian guy in particular. ‘Let me tell you, I’d walk into Brian’s office and say, Brian, you really messed that language up...’ The entire semester I was like who the f is this Brian guy? Years later I was like oh shit, Brian is Brian Kernighan, and the language is C! And I realized I missed a freaking awesome opportunity to ask Al Aho about some serious heads in CS.
- jandrese 8y agoPerl was basically written because Larry Wall found Awk's syntax to be a little too cryptic. In the language design business this is what we refer to as baby steps. Also, Awk isn't great for making reports, which is why Perl 5 to this day has an awkward report creation system[1] that looks like some COBOL refugee instead of idiomatic perl code. [1] https://perldoc.perl.org/perlform.html https://perldoc.perl.org/perlform.html
- setquk 8y agoUgh I’d forgotten about Perl reports. Gave up on that circa 2001 when I discovered python!
- kazinator 8y ago> Perl was basically written because Larry Wall found Awk's syntax to be a little too cryptic. That seems ridiculous; where is it substantiated? When Wall posted Perl to comp.sources.unix for the first time, he wrote "If you have a problem that would ordinarily use sed or awk or sh, but it exceeds their capabilities or must run a little faster, and you don't want to write the silly thing in C, then perl may be for you." Or rather, not Larry Wall, but the apparent newsgroup moderator added that text, lifting it from the Perl manual page. Thus he was pitching it as something that performs faster than awk and sed, with a greater range of capabilities.
- jandrese 8y agoThat was a little tongue in cheek but there is a comment in the Camel book about how he wrote perl because he was scared of Awk's parser.
- vram22 8y agoCould have been a joke. His writing (and talks, like the "State of" ones, is full of them, some of them a bit subtle too :)
- adrianmonk 8y agoOn pp. 381-382, my copy of Programming Perl (1992 printing of first edition) says he was trying to build a configuration management system for 6 Vax and 6 Sun machines, and he needed to solve some problems like file replication across a 1200 baud link and approvals. So he installed B-news, the Usenet news software at the time. Then he was asked to generate some reports and: > News was maintained in separate files on a master machine, with lots of cross references between files. Larry's first thought was "Let's use awk." Unfortunately, awk couldn't handle opening and closing of multiple files based on information in the files. Larry didn't want to have to code a special-purpose tool. As a result, a new language was born. So that's why it's the Practical Extraction and Reporting Language. He wanted to extract data from files and generate reports.
- nickdrozd 8y agoThe awk book is a lot of fun. I just finished reading it after seeing it recommended here a few weeks ago. The highlights for me: 1. A simple interpreter for an awk-like language called qawk. qawk is like awk except that it allows for querying by field name rather than field number. For instance, it allows doing { print $country, $population, $capital } instead of the more cryptic { print $1, $3, $5 } 2. An awk program that takes another awk program (in their example, a sorting algorithm) and outputs a version of that program modified to include profiling statements and an END section that outputs the results of those profiling statements to some file; then, another awk program that reads the data in that file and inserts that data back into the original awk program, thereby approximating where the hotspots are. There's a lot more in the book besides these, but to me these are the coolest programs because they are the awk-iest, by which I mean that they loop over lines of input, split the fields of those lines, and then manipulate the fields. Some of the programs in the book don't do this; instead, they consist of a single large BEGIN block with typical for-loops, arrays, etc. Used in this way, awk is just yet another dynamic language.
- dima55 8y agoGoogle doesn't know about qawk. A more general toolkit that does the same thing and (presumably) more: http://github.com/dkogan/vnlog http://github.com/dkogan/vnlog
- banku_brougham 8y agoThank you, I remember wanting to follow up on these more abstract constructions in the book. They seemed to be leading me somewhere amazing and very computer science-y. Programs that take programs as input and generate new code to do that thing I wanted with some data files — I’m sure this will be useful if I put the time in. Am I right that qawk was included as a program in the text? Did they ever follow up with further uses?
- nickdrozd 8y agoYeah, the code is all in the book, and it works! Here's the main body of the qawk interpreter: BEGIN { readrel("relfile") } /./ { doquery($0) } where - relfile is a file containing the field attributes used in various database files, - readrel is a function that parses the relfile and stores the fields in a dictionary, and - doquery is a function that takes a qawk query, converts it to an awk query by replacing the field names with their corresponding field numbers, and then executes the awk command. The whole thing runs about 60 lines.
- vram22 8y ago>If you are interested in learning Awk, I highly recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger. It's about the same size as the original "The C Programming Language" and is equally well-written. I'm a big fan of small utilities :) - as I sometimes say in my email sig; but more importantly, I'm a big fan of Kernighan et al, where by "et al" I mean the others from the core early Unix days, such as Dennis Ritchie, Rob Pike, Ken Thompson and many unnamed others, from whom I (and tons of others) learned about the Unix command-line (tools), the shell (scripting), and the Unix philosophy [1]. Had written this just a few weeks ago on HN, in the thread titled "Technical Writing: Learning from Kernighan", but worth repeating here in the context of this thread: https://news.ycombinator.com/item?id=17163276 https://news.ycombinator.com/item?id=17163276 It's a list of his books. I guess many may not know of some of them - I know I didn't. [1]: The Unix Philosophy in One Lesson: http://www.catb.org/esr/writings/taoup/html/ch01s07.html http://www.catb.org/esr/writings/taoup/html/ch01s07.html Attitude Matters Too: http://www.catb.org/esr/writings/taoup/html/ch01s09.html http://www.catb.org/esr/writings/taoup/html/ch01s09.html
- akavel 8y agoAs to one sentence in particular in ch01s09: "If someone has already solved a problem once, don't let pride or politics suck you into solving it a second time rather than re-using." I'd strongly argue it's overzealous. As much as I agree "reinventing the wheel" is dangerous, tempting, and can quickly spiral to yak shaving, but Unix itself, and all the good it brought, is a prime example of "solving [a problem] a second time" after Multics. In other words, I'd restate it in Sage Speak™: "Don't do this. Except when you need to." ;P Or, just want to have fun :P
- vram22 8y agoI sort of agree. I didn't quote those sections in the sense of recommending that the advice in them be followed strictly, and to the letter. Also, ESR is known to talk a bit that way - sort of overzealous, as you put it. But that is part of the fun of reading his stuff. As long as one takes it with a pinch of salt, and common sense, it is okay, and one usually gets to learn something from his writings.
- legends2k 8y agoI've heard about _The Awk Programming Language_ but could get my hands only on _The GNU Awk User's Guide_ by Arnold Robbins, since it's available as Info pages on my Linux box. It's also free online https://www.gnu.org/software/gawk/manual/gawk.html https://www.gnu.org/software/gawk/manual/gawk.html It's a pretty good book which teaches you both techniques and the nitty-gritty. I recommend this.
- marttt 8y agoI'm not a coder and awk was my first more serious or more complete effort to learn a programming language. What I did notice afterwards was that the syntax and semantics of C also became a lot more clearer to me. I think Kernighan also stated somewhere that awk was (also) designed as a helper-tool to learn C. All in all, I think it's a great first language, even if you're initially more compelled by Lisp family languages. If you have time to learn only one language, then awk is not a bad choice; It'll open many doors in the Unix/KISS-world. I can also state that "The Awk programming language" is, among other things, an excellent introduction to computer science or "the way programmers think" in general. A remarkably well written book for general audience.
- curiousgal 8y agoAwk is a neat language but the obsession of some programmers to condense awk commands into terse one-linners is just off-putting.
- nerdponx 8y agoThis, a hundred times over. If you actually sit down to write an AWK script, instead if trying to jam it into a one liner, you'll find that it's actually a fun, pleasant language for simple data processing tasks.
- digi_owl 8y agoE-peen swinging lives strong in IT, sadly. Just observe the kind of kindergarten antics some in the _sec world will get up to...
- kjeetgill 8y agoI don't want to speak for everyone, but that's a big part of it's value proposition: its brevity and ubiquity are very enabling. I don't think we'd be talking about awk right now if it wasn't useful inline in the shell. Plenty of people that would rather drop to Perl or Python once awk gets too cumbersome for the command line.
- mmt 8y agoAnd that's also much of what the article is about, though it's arguable if it's trying to get people more deeply interested in the language in general. Still, I'm not certain if the original comment is complaining about the tendency toward one-liners by actually writing less code (something the article alludes to, "only the inner part of the for loop") or merely cramming what would otherwise be readable as a multi-line, well-indented program into a single line.
- kazinator 8y agoSome terse Awk one-liners don't do anything clever; they just use common Awk idioms such as as { print } being the implied action if one is omitted and whatnot.
- 8y ago
- ams6110 8y agoI'm working on learning just a little perl. For some tasks it's amazingly effective, and it's usually installed even if Python isn't.
- lkurusa 8y agoJulia Evans (@b0rk) has a great zine about Awk: https://twitter.com/b0rk/status/1000604334026055681 https://twitter.com/b0rk/status/1000604334026055681 (The twitter thread that follows also has some great one-liners!)
- dominotw 8y agoProblem with this is that you keep relearning it every couple of months that you need it.
- ealhad 8y agoYou just have to need it more often ;) It can be really helpful in mundane data/logs analysis, for example.
- kjeetgill 8y agoI was in that boat for a long time and only have 4 or 5 key features of awk committed to memory now. If I have to look up something 2-3 times in a month it's usually committed to memory. Any less and it's probably better forgotten anyway, haha.
- dbolgheroni 8y agoSome people might think "why learn awk today when there is Python, Go, etc". But a lot of tasks (even big ones) can be done easier with this tool. The fact that most OSs have a POSIX compliant version of it in the base system also make it very valuable, not just for the people crunching data but also to sysadmin/devops people.
- digi_owl 8y agoAnd are probably less of a mess to get installed and running...
- Spivak 8y agoYeah, you can never be sure what version of a scripting language is going to be installed on a system. Always a little risky depending on the system version. These tools have the advantage of having not changed too much in the last few decades.
- jandrese 8y agoAwk works great in pipes on the commandline, unlike Python where you would be writing a multiline script to accomplish the same task. My 99% use case is extracting particular fields from some input stream. awk '{print $1,$5}' That's a tremendously common problem and awk is the best tool for the job. I once started diving deeper into awk and there is an impressive amount you can do with it, but I hit a point pretty quickly where it would be cleaner to write the script in a more traditional language where my successor won't be cursing my name for writing something complex in an obscure language.
- Spivak 8y agoIf you just want to extract fields the cut utility might be a better fit. cut -d $delim -f $field1,$field2,...
- mmt 8y agoI'll tend to do this in scripts, both for clarity and precision (i.e. when I know the specific, single delimiter). However, for ad-hoc command-line usage, I still prefer awk, because (1) the default delimiter is white space, which means I don't have to worry about if what I'm looking at might contain tabs intermingled with spaces (or how many spaces) and (2) it makes it much easier to incrementally increase the complexity of my ad-hoc operation, such as if I decide I want to take a sum or average of one of those columns, after all.
- ealhad 8y agoAwk is an amazing tool, and everyday I use it I wonder why I haven't learned it earlier.
- scbrg 8y agoI tend to post this whenever an awk discussion pops up, but at least for me, the reason I didn't learn awk earlier is that the gnu man page for awk is quite offputting. I spent ten years thinking "awk is too complicated for my weak mind." Then I stumbled on the man page of the plan9 implementation of awk and I learned the language in fifteen minutes - if even that. Sure, the gnu version is more powerful, but the plan9/original awk implementation handles most cases I run into, and I can now look up the gnu extras when I need them. http://man.cat-v.org/plan_9/1/awk http://man.cat-v.org/plan_9/1/awk
- coliveira 8y agoThis is a good point: GNU software complicates terribly what is supposed to be simple.
- ealhad 8y agoWhat can explain this?
- ealhad 8y agoYeah, in fact that's exactly the same reason for me. I remember when I started using GNU/Linux, trying to grok the most useful shell commands, and stumbling into awk... then quickly going somewhere else.
- dbolgheroni 8y agoawk is also interesting because it's not just a language, it's a "model". It isn't something that you can't write with some lines of any other language but, because the loop is embedded, makes it easier to write utilities for the command line.
- ollysb 8y agoYou can get the loop behaviour with ruby $ echo -e $'line one\nline two' | ruby -ne 'puts $_.split(/\W+/)[1]' one two
- fl0wenol 8y agoruby's -n and -p options are venerable equivalents to perl's options which were present in version 1.0 waaay back in 1987 and they were there so it could be that drop in replacement for awk ruby inherited a lot of structure and ideas from perl which it inherited from awk and that thought makes me irrationally happy
- Too 8y agoYou can easily wrap python with a small script that makes it more useful as command line filtering tool. Most of the bloat actually just comes from importing packages. #!/usr/bin/python3 import re, sys, json, other, useful, packages def printf(s): print(s.format(**globals())) for NR, line in enumerate(sys.stdin): match = eval(sys.argv[1]) if match: eval(sys.argv[2]) Then you can use this in command line as somecommand | pyawk.py "re.search(r'^(pattern)\s(otherpattern)', line)" "printf('{NR}) {match[1]}')" I deliberately didn't pass pattern straight into re.search but rather provides the option to do other forms of parsing/matching in that argument, for example json parsing. I just spent 2 minutes writing this script so it's obviously not perfect and you probably won't be able to use it for json out of the box but give it a few more iterations and it will most likely work.
- _31 8y agoI saw a comment just yesterday(?) on HN that mentioned awk and it got me interested (first time I heard about it). This was a good little intro that was easy to follow along with and get a sense of what it's used for.
- jedberg 8y agoAs a sysadmin, knowing awk, cut, and join is what lets you solve problems in seconds instead of minutes. I can't tell you how many times being able to quickly slice up a log file lead me straight to the problem I was trying to solve.
- j0e1 8y agoHere is an Awk program (not a one-liner) that extracted select snippets of text from lots of docs and worked almost 10x faster than a grep equivalent: awk 'BEGIN { ORS=" "} { print $0 }' $2 | awk --re-interval -v pat="$1" ' { cut_content = substr($0, 1, match($0, "==== Refs")) orig_content = substr($0, 1, match($0, "==== Refs")) idx = 0 regex = "(^|[^a-zA-Z]{1})" pat "([^a-zA-Z]{1}|$)" while (match(cut_content, regex)) { if (RLENGTH > 0) { prot = substr(cut_content, RSTART-1, RLENGTH+1) gsub(/[:punct: ]$/, "", prot) gsub(/^[:punct: ]/, "", prot) print prot, "\t", substr(orig_content, idx+RSTART-400, RLENGTH+800) idx += (RSTART + RLENGTH -1) cut_content = substr(cut_content, RSTART+RLENGTH) } } }'
- AdmiralAsshat 8y agoLooks interesting; can you explain how the hell it works? I tried writing a script that would extract entries from logs (e.g. if every entry started "====CRITICAL ERROR LOG 2018/06/15====", I wanted it to start there and grab every line below it until it hits the next log entry starting with the "=====" header), but I gave up. Your script might do something similar, if I can figure out what it's looking for.
- bcbrown 8y agoYou can use regex ranges to match every line between two regexes: https://www.gnu.org/software/gawk/manual/html_node/Ranges.html https://www.gnu.org/software/gawk/manual/html_node/Ranges.ht... Another option might be to use multi-line records (https://www.gnu.org/software/gawk/manual/html_node/Multiple-Line.html https://www.gnu.org/software/gawk/manual/html_node/Multiple-...). First pre-process the file to add an empty line before each header, then use blank lines as the record separator and newlines for the field separator.
- j0e1 8y agoSure. I'll explain the code and hopefully you'd find enough clues to adapt it for your use-case which, from what you've described, is a subset. Intent: Given a folder with multiple text files of academic papers from PubMed, remove the contents after the '====Refs' header. Then in each file find occurrences of any term in a keywords file and extract it along with 400 chars before and after it. Print this output as a tsv. input: $1) A text file containing keywords of interest that need to be searched in the files separated by `|`. $2) File containing paragraphs of lots of text with headers starting with '===='. There isn't a formatting requirement really. Just that, in my case the files happened to be so. # Note: Awk string commands have indexes that start with 1 (not 0). # The first Awk command just removes the newlines in file $2 and replaces them # with space. `pat` contains the contents of the file $1: 'keyword1|kewword2|...'. # --re-interval allows use of, well, regex intervals. awk 'BEGIN { ORS=" "} { print $0 }' $2 | awk --re-interval -v pat="$1" ' { # both variables are initialized with full-content of text file # with the part after '==== Refs' removed. cut_content = substr($0, 1, match($0, "==== Refs")) orig_content = substr($0, 1, match($0, "==== Refs")) # index (could be better named as offset, but life is too short to refactor). idx = 0 # match the keywords, even if it appears between non-alphabet chars. # e.g -keyword1. is a match. regex = "(^|[^a-zA-Z]{1})" pat "([^a-zA-Z]{1}|$)" # loop until matches are found. while (match(cut_content, regex)) { # RLENGTH is the length of the matched keyword. if (RLENGTH > 0) { # set the matched keyword to the variable `prot`. # RSTART is the character position of the matched keyword in the file. # This is the first occurrence of it in the file when read from the start. prot = substr(cut_content, RSTART-1, RLENGTH+1) # clean-up punctuation and/or space from the beginning and end of `prot`. gsub(/[:punct: ]$/, "", prot) gsub(/^[:punct: ]/, "", prot) # print the `prot` and 400 chars before and after it using idx # as an offset. Separate the results by tab char. print prot, "\t", substr(orig_content, idx+RSTART-400, RLENGTH+800) # update offset to character position right after the current matched keyword idx += (RSTART + RLENGTH -1) # cut-out all the text up until the position of the # last matched keyword (including it) and on the remaining text # repeat the above steps in the loop. This is so that the search # continues throughout the file and doesn't keep finding the same # keyword over and over again. See `while` condition. cut_content = substr(cut_content, RSTART+RLENGTH) } } }' Edit: Formatting. HN doesn't wrap text in code-block.
- kjeetgill 8y agoawk is irreplaceable on the command line because there's not any other good way to extract columns (awk '{print $2, $3}') or parse/work with numbers. How else would you print a running total for select columns easily? awk '$4>3 { x+=$3; print $2, $3, x}' It's the only tool that's aware of numbers, unlike grep and sed. My big new power move for awk is -F. You can use any regex as a field separator! Mind blown!
- v3gas 8y agoI made a tool for columns: https://github.com/vegarsti/csvprint https://github.com/vegarsti/csvprint
- sampo 8y ago> awk '{print $2, $3}' perl -lane 'print "$F[1] $F[2]"' > awk '$4>3 { x+=$3; print $2, $3, x}' perl -lane '$F[3] > 3 && {$x += $F[2], print "$F[1] $F[2] $x"}' Don't know about perl6, though. Edit: Better syntax per comments.
- kjeetgill 8y agoI like the awk better here but I'm sure there's loads more stuff perl could let me do that I can't here. Want to throw a few examples my way? Disclaimer: never learned perl
- acqq 8y agoAdd the "w" (for "warn") make it "-wlane" and read: Scalar value @F[0] better written as $F[0] at -e line 1. Scalar value @F[1] better written as $F[1] at -e line 1. In Perl 5 the array elements of @a are $a[0] $a[1] etc. However @F[0] works too because a "slice" is returned.
- lysium 8y agoShouldn’t that be $F[1] etc.?
- EvilTerran 8y agoOne trick I came up with recently: I found myself needing to add up a column of numbers, but there were so many that I started losing precision to awk's use of floats for all numbers. I wound up augmenting with `dc`[0] - something like (borrowing your example, and working from memory): awk ' BEGIN { print "0" } $4 > 3 { print $3"+p" } ' | dc [0] https://www.gnu.org/software/bc/manual/dc-1.05/html_mono/dc.html https://www.gnu.org/software/bc/manual/dc-1.05/html_mono/dc....
- linsomniac 8y agoI've never really been able to keep a significant amount of awk in my head. Mostly I use it for splitting out columns in output, especially when the fields may be padded for output. So "{ print $2 }". Awk has great abilities, but when I want to use it I always have to spend a lot of time reading. I've always wanted to come up with something in Python that combined the awesomeness of awk with my ability to pick up and hack something together in Python, but I've never been able to come up with the right semantics. One thing I think could be really useful is the regex lines range, where you give it two regexes and that block of code gets executed on every line in that range.
- Pete_D 8y agoNote that you can put conditions before the braces, so you can also express the example awk '{if ($(NF-2) == "200") {print $0}}' logs.txt as awk '$(NF-2) == "200" { print $0 }' logs.txt IMO, this style scans easier for most of the grep-ish use cases.
- kazinator 8y ago{ print $0 } is the default action if you don't specify one. Using "200" in quotes makes sense if you're looking for an exact string match; this will fail if the datum is 0200, or 200.0.
- newscracker 8y agoMy guess is that the GP's example is to get all HTTP status 200 lines from a web server log file. In that case, a precise match for "200" is fine.
- vram22 8y ago{ print $0 }, { print }, and nothing (i.e. no action given - but a pattern is given) are all equivalent, and result in printing the current line of the input (if it matches the pattern). Similarly, if no pattern is given, but an action is given, the default is to match every line (and so perform the given action on every line). Either the pattern or the action must be given, or both.
- cup-of-tea 8y agoMuch better. Awk is first a filter, then a processor. As others mentioned you don't even need a processor here. Using awk as just a filter is great and easily communicated by proper use of the language.
- kumarharsh 8y agoRecently, I was helping out a friend in analyzing some RNA samples for her work. These samples are huge - like nearly a gigabyte of data. There was this tool which was recommended for the job - mirexpress. It was a small job, perhaps 10 minutes worth of effort. To make my work easier, I provisioned a beefy (and costly) machine on Azure to do the job, took a quick look at the clock (it said 11 PM), ran the tool, and relaxed. The tool crashed while reading the file. In an attempt to fix the bug, I opened mirexpress's code. And all my confidence in my programming ability vanished when I saw its innards. I understand that the code may have been written by scientists who had no experience in programming, but I have never been so utterly _disoriented_ by bad code. Anyways, after hacking away at the mess for about 3-4 hours, I realized that this was a fool's errand and thought I'll just phone it in the next day saying I couldn't do it. I went to sleep thinking that it was already late and I'd get late for work the next day. - 5 minutes later - I woke up with a start, recalling this nifty tool called awk. I had last used it maybe 3 years ago, and before that only in college. But I could see how awk could do some of the things which mirexpress was claiming to do. So I fire up my computer, write an awk script - 2 lines only! TWO FUCKING LINES! And it runs like a charm - eats away at megabytes of sample data and gives me results I can show. So then like any rational person, I spent the remaining hours re-discovering awk and forgot to sleep. Pissed away the whole next day (and some part of the day after that too!) :-D It's really fascinating that this nifty little tools invented DECADES ago are still going strong, and there's been no _evolutionary_ leap in areas where tools like awk/grep/sed excel at.
- omginternets 8y ago>but I have never been so utterly _disoriented_ by bad code One of these days, when the mood strikes, I'll upload the matlab script I inherited from my Ph.D supervisor. Scientific programming is, by and large, an abysmal state of affairs.
- Waterluvian 8y agoAt one job I had we got "help" from a local college professor on how to design our database. A lot of what he said made sense. And we were super green and databases were not our expertise or focus. Short on time, high on cash, we asked if he'd implement it for us. I will never forget my gut wrenching feeling when I opened up one of the main provisioning scripts (in Python) to read the first two lines: True = 0 False = -1 The rest of it was an unmaintainable disaster that may have worked. I learned so much that summer, having been forced to learn it all myself.
- fizixer 8y agoshell + shell tools + piping = a league of its own. As a big fan and advocate of a low level language (C) and a high level language (Python), as a combo, being a very powerful paradigm, I still can't believe some of the things the, shall we call it, shell piping workflow, let's you accomplish in one line and in a matter of minutes. Here's what you do. Say you have a text log file. It's "semi structured" (like most log files) in that you can extract quite a bit of information using things like grep/sed/awk, but still not fully parse it (without a very complex parser; we don't want to get into that). What's the first thing you do with the log file? You view it: view logfile (view is just opening the file in vim in read-only mode, if you don't have it, you can use "vim -R"). You inspect the file and then quit. Try this instead: cat logfile | view - (don't forget the hyphen at the end). Again, inspect and quit (e.g., using :q). You might say, what a round-about/inefficient way to view a logfile. But. Now you can do something in the middle: cat logfile | do something | view - inspect and quit. Or more than one thing cat logfile | do something | do something else | view - and you keep adding the piped commands until you're satisfied with your output. Once you're satisfied, you can dump the output into another text file as a "report" by replacing the last "| view -" with "> reportfile". cat logfile | many | processing | commands | later > reportfile and you can do this multiple times to generate multiple report files of various kinds. And you can concatente some of those report files cat file1 file2 or put some columns of some of the report side by side paste -sd' ' <(cat file1 | pick column) <(cat file2 | pick column) The possibilities are endless. To give one example from my shell history. Some times I have a dozens of pdf files open in my linux desktop (using evince pdf viewer) and I wanna restart the computer but don't want to lose track of which pdf files were open. There may be automated ways of doing this but let's say there aren't any. I start with ps ax: ps ax | view - A long log. I want to pick only lines that have evince: ps ax | grep evince | view - Now I included the 'grep evince' line too, which I don't want: ps ax | grep evince | grep -v grep | view - Good. But I don't care about all the columns of ps log, except for the last one (the one that shows the full path of the file). That's column 6: ps ax | grep evince | grep -v grep | awk '{ print $6 }' | view - Looks good. Generate a report file from this: ps ax | grep evince | grep -v grep | awk '{ print $6 }' > openpdfs_YYYYMMDD.txt Then I close all my pdfs, and reboot. And I don't know grep, awk except for very basic things like what I already did. But I know similar tidbits about many other commands. If I have to perform regex search only I use grep, but for search and replace I use sed. Sometimes cut is more handy for column selection than awk. If I don't know the command but I know what I need to do, I just google it, and 99% of the time, I can find a stackoverflow post where a linux based one-liner is mentioned which is pretty much a drop-in for my piping workflow. Finally, if I want to traverse through each line of the entry and process one by one, I use the while loop. For example, if instead of dumping the file paths of the pdf I wanted to format it a little bit using directory name and file name, I'll do this: ps ax | grep evince | grep -v grep | akw '{ print $6 }' | while read -r dfnam; do echo 'Directory:' $(dirname ${dfnam}) 'File:' $(basename ${dfnam}); done | view - And again, view or dump. Try this, and you would realize that the possibilities are endless.
- inopinatus 8y agoIn this age where JSON is universally encountered, I find myself using jq(1) as much as I do awk. (1) https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
- coliveira 8y agoThe most important feature of awk is simplicity. No worries about complex language syntax, you just have a language that seems like C without everything that makes it complex. And it excels at a simple task: reading lines of data and transforming them as needed.
- oceanghost 8y agoDoes anyone else have trouble retaining the syntaxes, gotchyas, etc. from dozens of languages, hundreds of tools, 3 major operating systems... etc?
- jodrellblank 8y agoYes, which is one reason I like PowerShell so much. As a shell, the tab completion, parameter completion, long names, makes it easier to discover, easier to understand, and easier to remember. Then it's a high level language too so if there's something that needs scripting, it doesn't mean a complete change from shell to Python/Ruby/Perl, it stays PowerShell. Then it's a .Net language too, so if there's something getting a bit big for it, it doesn't mean a complete change to C#/Java instead, it means a small change to PowerShell with .Net methods, then maybe PowerShell with a C# core (like Python with a C module, but still much easier to create). Of course there's a bit of XKCD "fix having too many things by adding another thing" going on. But, the fact that it covers the common shell tools with all their different syntaxes reasonably well, and it can be tuned to approach the speed of C# as well, makes it useful for a whole lot of situations, despite having a pretty huge syntax and list of warts, it still seems to come out well.
- wyclif 8y agoPreviously (2011, the last time this was posted): https://news.ycombinator.com/item?id=2932450 https://news.ycombinator.com/item?id=2932450
- bajsejohannes 8y agoI once read the two or three first chapters of _ The AWK Programming Language_. Those chapters cover the whole language and I could read it in one sitting (actually standing; in a book store, but I digress). To my surprise, it was powerful enough to write a somewhat limited implementation of Snake: https://github.com/johshoff/snawk https://github.com/johshoff/snawk
- rbc 8y agoI mostly think of awk as a query language for text that's in record format. It doesn't seem to handle unformatted text very well and I tend to use sed or some other kind of tool to clean it up before feeding it to awk. I like the baked in logic for dealing with record/field formatted data. One thing that seems to hang people up is the lack of a concatenation operator. You just put things next to each other. It looks a little weird. Having associative arrays is a nice plus-up from shell programming.
- RickJWagner 8y agoHacker News paydirt! Awk is a great tool. Glad to see this.
- zeptomu 8y agoDon't forget about `-F`, the field delimator (which is WHITE by default). Obviously an option of awk, but some people disregard some *nix tools, because there data is not space delimited (and delimiter choice flags differ between tools :( ). For quick filtering it's really great, e.g. you can parse simple (pretty-printed) XML tags via `-F[<>] '{print $2}'` (for a quick glance on the data - of course not a good idea in production).
- skanga 8y agoAlso try mawk - it runs much faster than awk.
- mattbillenstein 8y agoI forget where I saw it now, but I've seen a pretty good write up on using unix tool to do very common operations -- set counting, histograms, etc -- and awk is at the heart of most of them. Usually with sort and uniq -c in there. It's amazing how quickly you can pick apart a log file -- minutes -- while it might take others hours to get to the same information using Excel.
- js2 8y agoI once wrote a network calculator in awk: https://gist.github.com/jaysoffian/e41ca479d70e60efe59fded93528ab1b https://gist.github.com/jaysoffian/e41ca479d70e60efe59fded93... Backstory: circa 2000, at an early cloud company that no longer exists, we used Solaris boxes and provisioned them using JumpStart. They booted to a minimal state (we called them "embros") using DHCP. Later, our datacenter ops folks would have to switch their networking from DHCP to static as part of moving them to another network where a DHCP server didn't exist. So I wrote them this menu driven program they could run on console to help with the static configuration. (There were no high-level languages on the box save for shell, sed and awk. Even then, I apparently I had to use /usr/xpg4/bin/awk.)
- noisy_boy 8y agoThat reminds me of the xpg extension of programs on Solaris boxes - I don't remember the specifics but often if I was looking for something that wasn't in /usr/bin/* I would check for the xpg version.
- js2 8y agoxpg4 is the X/Open Portability Guide Issue 4 versions of the programs. The commands in /usr/bin on Solaris were (are?) ancient backwards compatible versions.
- djtriptych 8y agoOne of the best things I ever did for my programming career was read The Linux Programming Inrerface cover to cover. Obviously I didn’t expect to completely grok 1500ish pages of tech details, but a survey of how Linux is organized and what is does well was a revelation. As a result, I’m WAY less likely to reinvent the wheel. I love to argue down tech proposals at work with something along the lines of “Linux already does that for you”. This sort of understanding can easily save man-years of effort on moderately complex undertakings. As I’m writing this I again wonder if some sort of professional licensing is needed in software engineering. A bioinformatics PhD has absolutely no way of assessing potential CS consultants, and picking the right one could save so much money, time, and effort.
- b0rsuk 8y agoMy boss recently asked me to write a couple of very simple Python scripts for mobile translations. I had to convert files to replace 𞉀 type sequences into proper UTF-8 codepoints. I had to replace a bunch of unicode codepoints with a mirrored en-us side. I had to convert lines like "abc" = "def"; into a trivial .json file (a single flat dictionary). The task was very simple in Python, but would be absolutely trivial and fun in AWK if it a) had first class unicode support, including conversion b) had a support for quoted fields. I think it needs a new variable, like QC (Quote Character) of FD (Field Delimiter). I would put money on a kickstarter of another crowdfunding initiative to modernize AWK. I don't mean by slapping a Python or other programming language on it, but by fishing long-standing issues with it. I think a for loop like in Python, Rust and VimL would be a better fit to an otherwise simple language (only C-style fors are available in BEGIN/END).
- hawski 8y agoWhat do you mean by C-style for? for (key in array) is POSIX. I miss in awk mainly a structural regular expressions mode. With it I would not miss lack of structures or that functions like gsub do not return a string instead of assigning it to a variable.
- b0rsuk 8y agoI mean for(x=0;x<10;x++). This stands out like a sore thumb in a language famous for its brevity and frictionless syntax. There should be a better way.
- n4r9 8y agoDon't know about the original awk but in gnu awk you have simpler for loops. https://www.gnu.org/software/gawk/manual/html_node/Scanning-an-Array.html https://www.gnu.org/software/gawk/manual/html_node/Scanning-...
- ElijahLynn 8y agoJust went through the whole thing, which was maybe 15-20 minutes of tinkering, and was way better than procrastinating on just reading other crap. Leveled up for sure, and it was well worth the 20 minutes!
- quickthrower2 8y agoI wonder how many Google job applications he's got in the last 14 hours!
- HeadlessChild 8y agoAccording to the footer he is a software engineer at Google.
- quickthrower2 8y agoYes, I am referring to > If you are the type of person interested in Awk, you are probably the type of person I'd like to see working with me at Google. If you send me your resume (____@gmail.com)
- indentit 8y agoawk is a great tool, and I used to use it quite a lot. But now that we have cross platform PowerShell, I find it so much easier to use that, its much more powerful
- slx26 8y ago>> To this day, 90% of the programmers I talk to have never used awk. Ok, time for someone that doesn't regularly use awk to say something. I understand that awk is great. The language being terse is nice if you use it regularly, but otherwise it's very easy to forget, and it's strange to look at it the first time. 90% of the programmers probably don't regularly need the kind of functionality awk provides, and the few times they do, they can create a simple script in a language they know better. Awk can be a super useful tool, but I think it's reasonable that most programmers don't use it, and the language is not designed to be used by everyone. In my opinion, a more universal alternative to this would be something like a web application that allows visual programming and translates the operations to awk, and possibly other languages too (the visual language could encode only a subset of awk, only common operations). You would learn awk yourself if you use it regularly, and you would know that you can simply use an intuitive interface to solve those formatting problems you need to handle from time to time otherwise.
- akavel 8y agoOk; my favorite awk one-liner I found somewhere and learnt by heart is for extracting "blocks of text between some starting pattern/line and some ending pattern/line"; most often, that would be "functions with particular name, with their whole body". Brace yourselves: awk '/^func Test/{p=1}; p; /^}/{p=0}' EXPLANATION First of all, this is 3 separate "commands", separated by ';'. In order: /^func Test/ {p=1} — if line matches regexp '^func Test' (i.e., starts with "func Test"), then set variable p to 1 (a.k.a. "True"). p — equivalent to any of: p { print } p { print $0 } { if (p) { print $0 } } meaning: if variable p is true-ish (in case of this script, if p==1), then print current line (if action is not specified after a condition, then it's by default {print}). /^}/ {p=0} — you may have guessed already now: stop printing after encountering end of function (line starting with '}').
- dbolgheroni 8y agoYou can get the same behaviour with sed without "programmability": $ sed -n '/^func Test/,/^}/p' file Which means: match 'func Test' at the beginning of a line and print (p command at the end) until you find a line beginnning with '}'. This is because the print command (p) accepts 2 addresses to delimit a range. I used regexes as addresses but, for instance, can use line numbers also: $ sed -n '10,20p' file This prints file's line from 10 to 20. By default, sed prints the pattern space (modifications to each line) at the end of the script. The -n I used is to avoid that.
- akavel 8y agoHah, lol, thanks, Today I Learnt ;) Will try using this next time, probably. That said, there's one extra advantage with the awk script, that by rearranging the expressions appropriately, I can choose whether to include or exclude the the first and last line in the output (i.e. 'FIRST; p; LAST' vs. 'FIRST; LAST; p' or 'p; FIRST; LAST', etc.) Is this also possible with sed? :)
- forinti 8y agoAwk is nice to know, but my experience is that Perl one-liners are a lot faster. And where you have Awk, you usually find Perl also.
- artisin 8y agoI'm a big fan of "one-liners". Granted they're not always the best solution but they're hard to beat. Something like printing the disk space across filesystems: df -m | awk '{p+=$3}; END {print p}' I bought a book titled "Awk One-Liners Explained" on a whim a while back. And to this day, I consider it to be umong the most "useful" $6 I've ever spent in terms of productivity.
- u801e 8y agoI used to write scripts with awk, but stopped doing that after learning perl. I wonder if learning how to use perl with the -ane option would be easier compared to learning awk (especially in terms of text maninpulation).
- starchild_3001 8y agoAs an ML engineer & former data scientist, I use awk almost everyday. It's indispensable to pick columns, run simple stats, parse unstructured files, convert between two formats... Hacky, but often the fastest tool.
- olskool 8y agoBack in the 1980s my company wouldn't buy me a database so I hacked one up in awk.