3 ms·
sed and awk complement each other, I think. sed works on a line by line basis. awk can work on a whole file. Subsequent line operations can depend on the stat
by sohkamyung 9y ago
sed and awk complement each other, I think.
sed works on a line by line basis.
awk can work on a whole file. Subsequent line operations can depend on the state of previous lines.
Each has its own operating domain and you have to decide which tool is the best one for the task you have in mind.
- padthai 9y agoI use awk every day because I need state (I work with text files full of sections and subsections) but I am sure that there has to be something better out there. What is the definitive tool to process text? Perl? Haskell? Some Lisp dialect?
- rlonstein 9y ago> What is the definitive tool to process text? Perl? Haskell? Some Lisp dialect? Definitive? Being snarky, the one you have already installed and are familiar with. Like most I use Awk for one-liners, Perl if I need a little more or better regexes in a one- or two-liner. For the last several years I've been using TXR[1] if it gets complex. Lately I've been doing more fiddling with JSON than text and I'm using Ruby/pry and jq[2]. [1] http://www.nongnu.org/txr/ http://www.nongnu.org/txr/ [2] https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
- kazinator 9y agoHi; I replied to your Github gist quite a while ago: https://gist.github.com/rlonstein/90d53fdeea31d2137737 https://gist.github.com/rlonstein/90d53fdeea31d2137737 about a matter related to the hash bang line in the script. TXR has a nice little hack (that apparently I invented) to implement the intent of "#!/usr/bin/env txr args ..." on systems where the hash bang mechanism supports only one argument after the interpreter name.
- macintux 9y agoBiased because I've used Perl for over 20 years, but yeah, that's clearly one of its core reasons to exist. Regular expressions built into the language syntax instead of as a library makes a big difference.
- znpy 9y agoPerl. It was designed for doing just that :)
- Annatar 9y agoPerl has an intangible "write once" property: since it allows for writing extremely sloppy code under the "there is more than one way to do it!" mantra, nobody, including the original author can debug it afterwards. Not even with the built in Perl debugger. Perl encourages horrible spaghetti code. In the interest of fair and accurate disclosure, I earned my bread for 3.5 years debugging Perl code for a living and I've also had formal education in Perl programming at the university. I would never want to do that again.
- majewsky 9y agoI have spent my first 2.5 years out of college working on legacy Perl code and I cannot agree. Perl is a very nice language if you follow a coding style, and really, any language gets ugly pretty quickly if you don't. There's this adage that "some developers can write C code in any language", and it's probably similarly true that some developers can write Perl one-liners in any language. (In that legacy Perl codebase that I maintained, one of the devleopers was clearly writing Fortran code in Perl. He was doing everything with nested loops over multi-dimensional integer arrays.)
- macintux 9y agoI experimented a bit with writing Erlang-style code in Perl. Wasn't terribly successful; pattern matching, even with regular expressions built into the language, is a fairly tough feature to emulate.
- kbenson 9y agoThe problem is that with regexps you're generally still doing text matching, which is inefficient and error prone. Perl's default exception mechanism allows text based errors as well, so you end up doing it there too if you use exceptions and haven't decided on and strictly used some exception objects by default (and even then you either need to deal with strings as you encounter them, such as promoting them to an exception object of some sort). Objects at least allow you to definitively match on types. Perl's (now) experimental given/when constructs and smartmatch operator would help with this, but they've been plagued with problems for a long time now (or at a minimum are not seen as production ready still).
- Xophmeister 9y agoIndeed: sed is useful for making small, line-wise tweaks to text. To be honest, I use it rarely (and this is largely because its regexp flavour leaves a lot to be desired). Things like delete the header line[1S] or a simple replacement[2S]. It, like Awk, has some useful line targeting functions (e.g., print lines between two regular expressions[3S], etc.) Awk, on the other hand, is more like a finite state machine for text processing, with the notion of records and fields baked in[4]. You can do the same thing in Awk as in sed (see [*A] references), but it's often easier in sed; vice versa, some things would be impossible or very difficult to do in sed which would be easy in Awk (e.g., [4], which prints the fifth field whenever the first field is "foo"). This doesn't even get into the multiline/statewise stuff you can do in Awk, but the examples would be too big/specific to fit into this comment. I also learned recently that GNU Awk has networking support[5]. I have no idea why! [1S] sed '1d' [1A] awk 'NR!=1 {print}' [2S] sed 's/foo/bar/' [2A] awk '{sub(/foo/, "bar")}' [3S] sed -n '/start_regex/,/end_regex/p' [3A] awk '/start_regex/,/end_regex/ {print}' [4] awk '$1=="foo" {print $5}' [5] https://www.gnu.org/software/gawk/manual/gawkinet/gawkinet.html https://www.gnu.org/software/gawk/manual/gawkinet/gawkinet.h...
- fosco 9y agoI did not know gawk had networking support I wonder if it could be used on network traffic on the fly. Sort of like irules on an f5. Thank you for sharing!
- vram22 9y ago[3A] awk '/start_regex/,/end_regex/ {print}' can be simplified to: awk '/start_regex/,/end_regex/' because in awk, if no action is given, the default action is to print the lines (that match the pattern). And if the pattern is omitted but the action is given, it means do the action on all lines of the input. Edited to change: print the line (that matches the pattern) to print the lines (that match the pattern)
- lo_stronzo 9y ago"sed is useful for making small, line-wise tweaks to text." Couldn't agree more! A great example of this was using (surprised it wasn't mentioned) sed with the -i & 's///g' operators while "cleaning" hundreds (seriously) of HTML/PHP files from injected content at a shared hosting provider.
- bananicorn 9y agoHonestly, that makes sense - doing multiline replaces with sed isn't very convenient (I believe it's possible if you replace newlines with NULL). I guess I'll probably learn awk then, it can't be that hard with the examples from this repo^^