7 ms·

I'm wondering: Does awk really provide that more value over sed while being easier or faster to use than a fully-fledged scripting language (thinking of perl,

• __MatrixMan__ 9y ago
  I'll admit that I don't know perl, which I believe is the typical sed+awk replacement. According to my potentially miscalibrated gut, referencing a "fully-fledged" scripting language in a shell script or at the command line is an indication that you should probably just be working in that environment in the first place. There will be exceptions, it depends chiefly on the problem being solved, but overall I prefer my shell scripts to reference utilities with very specific purposes. It feels UNIXier that way.
• sohkamyung 9y ago
  sed and awk complement each other, I think. sed works on a line by line basis. awk can work on a whole file. Subsequent line operations can depend on the state of previous lines. Each has its own operating domain and you have to decide which tool is the best one for the task you have in mind.
  • padthai 9y ago
    I use awk every day because I need state (I work with text files full of sections and subsections) but I am sure that there has to be something better out there. What is the definitive tool to process text? Perl? Haskell? Some Lisp dialect?
    • rlonstein 9y ago
      > What is the definitive tool to process text? Perl? Haskell? Some Lisp dialect? Definitive? Being snarky, the one you have already installed and are familiar with. Like most I use Awk for one-liners, Perl if I need a little more or better regexes in a one- or two-liner. For the last several years I've been using TXR[1] if it gets complex. Lately I've been doing more fiddling with JSON than text and I'm using Ruby/pry and jq[2]. [1] http://www.nongnu.org/txr/ http://www.nongnu.org/txr/ [2] https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
      • kazinator 9y ago
        Hi; I replied to your Github gist quite a while ago: https://gist.github.com/rlonstein/90d53fdeea31d2137737 https://gist.github.com/rlonstein/90d53fdeea31d2137737 about a matter related to the hash bang line in the script. TXR has a nice little hack (that apparently I invented) to implement the intent of "#!/usr/bin/env txr args ..." on systems where the hash bang mechanism supports only one argument after the interpreter name.
    • macintux 9y ago
      Biased because I've used Perl for over 20 years, but yeah, that's clearly one of its core reasons to exist. Regular expressions built into the language syntax instead of as a library makes a big difference.
    • znpy 9y ago
      Perl. It was designed for doing just that :)
      • Annatar 9y ago
        Perl has an intangible "write once" property: since it allows for writing extremely sloppy code under the "there is more than one way to do it!" mantra, nobody, including the original author can debug it afterwards. Not even with the built in Perl debugger. Perl encourages horrible spaghetti code. In the interest of fair and accurate disclosure, I earned my bread for 3.5 years debugging Perl code for a living and I've also had formal education in Perl programming at the university. I would never want to do that again.
        • majewsky 9y ago
          I have spent my first 2.5 years out of college working on legacy Perl code and I cannot agree. Perl is a very nice language if you follow a coding style, and really, any language gets ugly pretty quickly if you don't. There's this adage that "some developers can write C code in any language", and it's probably similarly true that some developers can write Perl one-liners in any language. (In that legacy Perl codebase that I maintained, one of the devleopers was clearly writing Fortran code in Perl. He was doing everything with nested loops over multi-dimensional integer arrays.)
          • macintux 9y ago
            I experimented a bit with writing Erlang-style code in Perl. Wasn't terribly successful; pattern matching, even with regular expressions built into the language, is a fairly tough feature to emulate.
            • kbenson 9y ago
              The problem is that with regexps you're generally still doing text matching, which is inefficient and error prone. Perl's default exception mechanism allows text based errors as well, so you end up doing it there too if you use exceptions and haven't decided on and strictly used some exception objects by default (and even then you either need to deal with strings as you encounter them, such as promoting them to an exception object of some sort). Objects at least allow you to definitively match on types. Perl's (now) experimental given/when constructs and smartmatch operator would help with this, but they've been plagued with problems for a long time now (or at a minimum are not seen as production ready still).
  • Xophmeister 9y ago
    Indeed: sed is useful for making small, line-wise tweaks to text. To be honest, I use it rarely (and this is largely because its regexp flavour leaves a lot to be desired). Things like delete the header line[1S] or a simple replacement[2S]. It, like Awk, has some useful line targeting functions (e.g., print lines between two regular expressions[3S], etc.) Awk, on the other hand, is more like a finite state machine for text processing, with the notion of records and fields baked in[4]. You can do the same thing in Awk as in sed (see [*A] references), but it's often easier in sed; vice versa, some things would be impossible or very difficult to do in sed which would be easy in Awk (e.g., [4], which prints the fifth field whenever the first field is "foo"). This doesn't even get into the multiline/statewise stuff you can do in Awk, but the examples would be too big/specific to fit into this comment. I also learned recently that GNU Awk has networking support[5]. I have no idea why! [1S] sed '1d' [1A] awk 'NR!=1 {print}' [2S] sed 's/foo/bar/' [2A] awk '{sub(/foo/, "bar")}' [3S] sed -n '/start_regex/,/end_regex/p' [3A] awk '/start_regex/,/end_regex/ {print}' [4] awk '$1=="foo" {print $5}' [5] https://www.gnu.org/software/gawk/manual/gawkinet/gawkinet.html https://www.gnu.org/software/gawk/manual/gawkinet/gawkinet.h...
    • fosco 9y ago
      I did not know gawk had networking support I wonder if it could be used on network traffic on the fly. Sort of like irules on an f5. Thank you for sharing!
    • vram22 9y ago
      [3A] awk '/start_regex/,/end_regex/ {print}' can be simplified to: awk '/start_regex/,/end_regex/' because in awk, if no action is given, the default action is to print the lines (that match the pattern). And if the pattern is omitted but the action is given, it means do the action on all lines of the input. Edited to change: print the line (that matches the pattern) to print the lines (that match the pattern)
    • lo_stronzo 9y ago
      "sed is useful for making small, line-wise tweaks to text." Couldn't agree more! A great example of this was using (surprised it wasn't mentioned) sed with the -i & 's///g' operators while "cleaning" hundreds (seriously) of HTML/PHP files from injected content at a shared hosting provider.
  • bananicorn 9y ago
    Honestly, that makes sense - doing multiline replaces with sed isn't very convenient (I believe it's possible if you replace newlines with NULL). I guess I'll probably learn awk then, it can't be that hard with the examples from this repo^^
• cyborgx7 9y ago
  To me, awk always occupied an awkward (no pun intended) spot. Too complicated for a proper single purpose command-line program. Not expressive enough to be in the same sphere as a scripting language like the ones you mentioned. I was hoping this article might illuminate its purpose, but I'm still just as clueless.
  • asicsp 9y ago
    could you give an example use-case? >Too complicated please try out examples given and let me know if it helps you to understand the syntax better
• oblio 9y ago
  awk is my go-to tool for simple command line tokenizer. Hard to beat: awk '{ print $1; }' Other than that... not really. Maybe the advantage would be ubiquity, if you really, really want to avoid Perl.
  • cyborgx7 9y ago
    >Hard to beat: awk '{ print $1; }' How about: cut -f 1 -d ' ' You don't even need the -d flag if the you happen to be able to use the default delimiter, like in your example.
    • zb 9y ago
      I used to regularly do ad-hoc text processing, typically on a 24 hour log of GPS data (on an early-2000s-era computer). Surprisingly enough, awk is many times faster than cut for any data set big enough for you to notice time passing.
    • deleted 9y ago
      [deleted]
    • fnord123 9y ago
      Awk delimits on whitespace by default. cut cannot do that afaict. So if you have something like this cut wont work: apples 1 bananas 2
      • ninkendo 9y ago
        Exactly, which makes cut kinda shit for 99% of common "split-on-whitespace" tasks IMO.
      • dredmorbius 9y ago
        You can always use awk with IFS/OFS values to clean up the delimiters so you can pass the data off to cut ;-)
  • sbmassey 9y ago
    There's always plain old bash while read a _; do echo $a; done
• mohaine 9y ago
  If I remember correctly, I believe perl was created to combine the power of sed and awk into a single tool so you didn't have keep switching back and fourth between the two. Personally, I would suggest AWK for one to two liners when doing some one time data transform tasks. Anything more complicated and I would suggest a more "fully-fledged scripting language"
• brians 9y ago
  For me, yes. I use Python plenty too, but Awk is great for a tiny state machine, and unique in the ease of setting up an event stream view of a text file. I could use Python, add a loop and a big set of conditionals—but a one page awk program gets all that right.
• asicsp 9y ago
  * for search and replace on entire lines and other address based filtering, I prefer sed (or perl if I need PCRE features like non-greedy, lookarounds, code in substitution section, etc) * for field processing, most multiple line processing, use of logic operators, arithmetic, control structures etc, I prefer awk or perl * this repo is aimed at command line text processing tools, most awk examples given are single line, a few are 2-3 lines. Personally I prefer Python for larger programs ---- See also: https://unix.stackexchange.com/questions/303044/when-to-use-grep-less-awk-sed https://unix.stackexchange.com/questions/303044/when-to-use-...
• abecedarius 9y ago
  If you already know it, then yes, it's still handy sometimes. And it's a small language, unlike Perl. OTOH I'm not sure I'd recommend bothering to learn it. Python is more verbose in Awk's domain, but not by so much as makes a huge difference, except at the scale of one-liners. (Or a-few-liners, at least.) Another reason to learn it: the AWK book (by Aho, Kernighan, and Weinberger) is a great very short intro to the spirit of Unix-style coding. You could think of learning Awk as just the price of admission to that intro, paid along the way. I wrote plenty of Awk in the 90s -- https://github.com/darius/awklisp https://github.com/darius/awklisp isn't very representative but it was fun.
• cup-of-tea 9y ago
  It's more powerful than sed while being much easier and quicker to use than perl or python, so yes. I use it quite often just as a filter. sed is still good for some things.
• Annatar 9y ago
  AWK is a programming language. In the AWK book by Aho, Weinberger and Kernighan, towards the end of the book they implement an arbitrary assembler, the virtual processor and a virtual machine for the machine code they just invented to run. They also implement a relational database management system in AWK, as well as an autoscale graphing solution. I myself have implemented XML SOAP command line client, a backup solution, a SAN UUID management application and an automated Oracle RAC SAN storage migration solution, a configuration management, and an Oracle database creation / management applications in AWK. Usually I develop a thin getopts shell wrapper around an AWK core. Works every time, the executables are on the order of a few KB (the largest so far, the XML SOAP client is 24.5 KB) and they all run like a bandit. Memory requirements are miniscule. Dependencies are minimal: the only external dependency so far in my software has been the xsltproc binary from the libxslt package. AWK is easier to use than Python or Perl, and is much faster than either of those. Typical code density ratio of Python versus AWK is 10:1, sometimes more. This means that if you have a 650 line Python program, you can implement the same functionality in about 280 lines of AWK, and the program will be far simpler. I've once collapsed a 280+ line Python program into a simple 15 lines of code in AWK. AWK is an extremely versatile, powerful programming language. For even more speed, AWKA can be used to transpile AWK source into C and then it will call an optimizing C compiler to compile it into a binary executable. Typical speedup is on the order of 100%, so if your AWK program ran in 12 seconds, it'll now finish in six.
  • segmondy 9y ago
    I once introduced AWK to a team and shared the same book with them. They didn't know that AWK was a programming language. Told them they could achieve more faster with AWK than using python & php for transforming and shuffling around data. They looked at me like I was crazy. :(
  • zephyrfalcon 9y ago
    """Typical code density ratio of Python versus AWK is 10:1, sometimes more. This means that if you have a 650 line Python program, you can implement the same functionality in about 280 lines of AWK, and the program will be far simpler. I've once collapsed a 280+ line Python program into a simple 15 lines of code in AWK.""" How does this work? I am not saying it can't be done, but the main benefit of Awk seems to be quick one-liners, which are possible because you get "records" (splitting on whitespace) and lines (splitting on newline) and looping for free. But for larger programs, this easily translates to Python; just call readlines(), loop over it, call split() on each line. I would think that at this point, Awk doesn't have much of an advantage anymore... but apparently your experiences are different. What are some Awk constructs that would take a lot more code in Python?