3 ms·

For many years my AWK knowledge was limited to basic '{print $1}' style usage. I never bothered to learn more. I tended to use perl when I needed a custom text-

For many years my AWK knowledge was limited to basic '{print $1}' style usage. I never bothered to learn more. I tended to use perl when I needed a custom text-processing operation. Later, as perl became less a part of my working life, I began using ruby instead - they are pretty similar in spirit. One day, nearly 20 years after it was published, I picked up a used copy of The AWK Programming Language by Aho, Kernighan, and Weinberger. Yes, they are credited in that order on the cover... I suspect intentionally. I only read the first N chapters, but it was enough. I used AWK many times within the following month, and I continue to use AWK on a daily basis. When the task is complicated, I will still use ruby, but often enough AWK is easier. The point: you think "Why would I learn X when I can use Y?", but you won't really know the answer until you learn X. If I had never learned perl, python, ruby, AWK, shell script, vi macros, then I would probably be editing files by hand (!) like I sometimes catch developers actually doing (!!!). For a person who doesn't know these tools, that might actually be the path of least resistance. Investing some time here and there to learn new tools pays off in the future in ways that are unpredictable.

jacobolus 7y ago
The basic imperative Python version is much easier to remember and read though, even for not-that-experienced Python programmers. I would expect laypeople to be able to more-or-less figure out what it is supposed to do. seen = set() with open(filename, "r") as file: for line in file: if line not in seen: print(line) seen.add(line) Often (at least in my experience) this kind of operation is either (a) part of some larger automated data processing pipeline for which it’s really nice to have version control, tests, ... or (b) part of some interactive data exploration by a programmer sitting at a repl somewhere, not just a one-off action we want to apply to one file from the command line. In those contexts, the Python (or Ruby or Clojure or whatever general-purpose programming language) version is easy to type out more-or-less bug-free from memory, debug when it fails, slot into the rest of the project, modify as part of a team with varied experience, etc. etc.
- skinner_ 7y ago
  One advantage is that seen.add(line) can be changed to seen.add(hash(line)) which can be significantly more memory efficient for files with long lines.
  - jacobolus 7y ago
    Or perhaps better, if needs change the seen = set() object can be swapped out for any alternative object seen = foo that provides foo.__contains__ and foo.add methods. This could involve saving previously seen lines in a radix tree, adding multiple layers of caching, saving infrequently seen lines to disk or over the network, etc. as appropriate for the use case.

jacobolus 7y ago

The basic imperative Python version is much easier to remember and read though, even for not-that-experienced Python programmers. I would expect laypeople to be able to more-or-less figure out what it is supposed to do. seen = set() with open(filename, "r") as file: for line in file: if line not in seen: print(line) seen.add(line) Often (at least in my experience) this kind of operation is either (a) part of some larger automated data processing pipeline for which it’s really nice to have version control, tests, ... or (b) part of some interactive data exploration by a programmer sitting at a repl somewhere, not just a one-off action we want to apply to one file from the command line. In those contexts, the Python (or Ruby or Clojure or whatever general-purpose programming language) version is easy to type out more-or-less bug-free from memory, debug when it fails, slot into the rest of the project, modify as part of a team with varied experience, etc. etc.

skinner_ 7y ago
One advantage is that seen.add(line) can be changed to seen.add(hash(line)) which can be significantly more memory efficient for files with long lines.
- jacobolus 7y ago
  Or perhaps better, if needs change the seen = set() object can be swapped out for any alternative object seen = foo that provides foo.__contains__ and foo.add methods. This could involve saving previously seen lines in a radix tree, adding multiple layers of caching, saving infrequently seen lines to disk or over the network, etc. as appropriate for the use case.