7 ms·

I really dislike sed, awk, and all these utilities because I just can’t remember how to use them. Every time I want to use one of these tools I have to relearn

by baby 5y ago

I really dislike sed, awk, and all these utilities because I just can’t remember how to use them. Every time I want to use one of these tools I have to relearn the stuff, and then after a while I forget how to use it. If I don’t use it all the time I just forget. I really don’t think non-interactive CLIs are a good way to do complicated tasks.

Debug_Overload 5y ago
How long does the relearning take exactly? My experience has been that if you've used these tools before, it's pretty easy to remember or refresh your memory. In that sense, they're not very different from other tools/syntaxes that you haven't used in a while. And if you do use them frequently enough, you might not even need to refresh/relearn anything.
- baby 5y ago
  I basically gave up on relearning them considering the number of times I’ve had to do it and the amount of effort involved to relearn them. Maybe it’s just me, but I just write python now.
  - sam_lowry_ 5y ago
    You need to have a mental model for each tool and some idea of syntax associated with it. Once I learned about IFS,OFS, BEGIN, END and {}, as well as the fact that variables and their references have the same syntax, unlike in Perl... I could pick up coding in awk at any moment
    - chasil 5y ago
      IFS is the POSIX shell; in awk, it's just FS. I think by {} you mean an argument to the find command.
      - sam_lowry_ 5y ago
        Indeed. Thanks for the hint.
    - kbenson 5y ago
      Perl has BEGIN and END as well, and when using it with -n or -p for line processing mode uses -F to set word boundaries just like awk. In fact, it shared many flags and features with awk, which it was meant to be able to be used as a more powerful version of the same tool.
  - ivanhoe 5y ago
    Makes sense if python is what you otherwise use for everyday work, it's always easiest if you can avoid context switching. To me it's bash, sed and awk for the basic text-processing things, and then perl for anything that requires more programming. These are very similar syntaxes (and ways of thinking I'd say), so they fit together great - and also are close enough to php and js that I do at work to make the transition painless. But if I had to write it in python or ruby or God forbid C++ - which all I used a lot, but long time ago - I'm sure it would be a big struggle for me now.
  - chasil 5y ago
    The full specification for the AWK language is 40 pages in the book written by the authors. I sometimes forget the exact syntax for pipes and file open/close, but it's not hard to find in 40 pages. AWK is in busybox, and a lot of other places that Python simply cannot go. The control structure syntax is also the same a C/JavaScript/PHP/C++ etc., so it certainly does not hurt to know it. https://archive.org/download/pdfy-MgN0H1joIoDVoIC7/The_AWK_Programming_Language.pdf https://archive.org/download/pdfy-MgN0H1joIoDVoIC7/The_AWK_P...
thaumaturgy 5y ago
I'm mostly with you (except for your last sentence). I've picked up a lot of languages over the years, and can fall back into most of them pretty fast, but sed and awk especially are hard. On the occasions that they really are the best tool for the job, I get actual books out and set aside a window of a few hours to plod through it. I think the problem is that they fall into an uncanny valley between "simple tool" and "verbose programming language"; they are really terse languages wearing a simple tool's sheepskin.
- sitkack 5y ago
  If I measure time to solve a problem and not line count, using a language like Python or Ruby might be 2x the lines , for a line count approaching zero anyways, not sure that matters. Performance is comparable and I don't need to relearn anything and someone else who doesn't know awk/sed doesn't need to learn it. And when one is using awk/sed that script is often invoked from a shell, bash for instance. And since the awk/sed can't hold its own as a general purpose programming language, now I am writing bash code, a personal sin. Awk/sed made sense for the time and place they are from, but things have changed.
  - thaumaturgy 5y ago
    Yes, this is a Standard HN Reply: the "Specialist's Rebuttal". It's usually in some form of, "___ is obsolete, ___ is better in every way, everyone should just be using ___ now." Okay, acknowledged. Python and Ruby are superior in every regard. Would you mind letting us chat about sed and awk now for a minute?
    - yesenadam 5y ago
      I replied to someone's "There is no reason for anyone to program in C nowadays" recently on here with "But I like programming in C". They hadn't considered that reason in their mental survey of Every Possible Reason. Also.. I use Awk every day, for all kinds of things. I forget the order of parameters in split, sub etc, which takes a few seconds to look up with man awk. I got into sed a while ago but never use it. Awk, together with sort and uniq sometimes, is all I need. And it is interactive. You have an input. Print that out and look what needs to be done. Write the first transforming step in Awk. See what that did. Add the next step. See what that did. Extremely interactive. After a few minutes I have a one-liner (often 2 or 3 wrapped lines) that does exactly what I want, and easy to adapt to other similar things. I have very large bash history enabled so they stay Ctrl-R-accessible.
aidos 5y ago
That’s just because you don’t use them often though. If they were a daily driver it’d be different. I think that if you have a good mental model of how awk works, then you know times that it’s the right tool for the job. And on those occasions you’ve got plenty of time to go search for some syntax, make a coffee, maybe even go for a run all with change to spare compared to using pretty much anything else. I will admit, that if I can do it with a few greps, seds and cuts then I tend to just use those first, because there no thinking involved for me.
ZeroGravitas 5y ago
I mostly find them useful when you can't be sure anything else will be installed. What we really need is something that can "compile" down to sed for deployment in that kind of environment. Then that compiler can contain all the weird stuff you need to remember.
s1291 5y ago
cheat.sh is your friend. E.g for sed: curl cheat.sh/sed Will show you several examples.
- News-Dog 5y ago
  cheat.sh & cht.sh < Nice! https://github.com/chubin/cheat.sh https://github.com/chubin/cheat.sh Also, LDP: Advanced Bash-Scripting Guide: https://tldp.org/guides.html https://tldp.org/guides.html
musicale 5y ago
Since sed is the stream version of ed/vi, it used to be easier to use and remember for people who used ed/vi as their primary editor.
mellavora 5y ago
awk is always a struggle for me to re-learn, but sed is simple. I basically use it for simple text substitutions; in that case it is the same as what I use for vim editing.
adrianscheff 5y ago
Haha, one of the reasons I wrote this. It's also a way of practicing and reminding myself sed.
OJFord 5y ago
Personally I'm fine with using a complicated expression (or a pipeline) built out of basic parts where it's not necessary (because there's some whizzbang operator that'll make it so much more succinct) if that's what I can remember or what is clearer to understand with fresh eyes.
superasn 5y ago
Which is why I've always stuck to `perl -pi -e`. It may not be as powerful but it's just one thing to learn. Also it understands regexs like '\d', etc which are kinda universal but for some reason don't work with many of these utils.
- mdaniel 5y ago
  I also read once that perl was the most ubiquitous tool across all Unixen flavors, but I don't have a link to cite for it
- gnubison 5y ago
  Perl is certainly more powerful than sed or awk :)
- kbenson 5y ago
  I wouldn't call having the de-facto standard for an advanced regular expression system, a full programming language and the entirety of the CPAN module system easily pulled in and used less powerful. I use sed for a lot of simple stuff, but I use Perl all the time for more advanced stuff. Perl is great for grabbing complex chunks out of a line and doing something with them that requires a bit of state. For example, writing a quick and dirty logfile parser to figure out how many requests and how many bytes were used by each IP over a time period (the time period limiting usually being a prior grep before being passed to Perl). I've probably taken 60 seconds to write out something to parse exactly what I want from a log hundreds of times.
zouhair 5y ago
In these instances where you find a one liner useful but you use it rarely, just put it in a function with comments on how it works. When you want to use it the function is ready and the explanation too.
memetomancer 5y ago
May I ask what is the alternative? Creating an entire application for a single use? designing a generalized case and making a monolithic application with lots of obscure options? What about setting up the project for the application, setting up instrumentation and testing, debugging, etc.? Even supposing that all that extra work makes sense for your unique string manipulation needs, what about the next time something like this comes up but is out of scope for your first tool? Do you end up with lots of little applications, with lots of little features and arguments? When you come across that similar-but-not-quite need down the road, do you have to scour your library of full blown apps to find something that fits? Not to mention maintaining them against future system updates... OR - you could just force yourself to get over the hump, learn the conceptual basics at play and then when you come across a novel situation down the line you can just rattle off a tailored one-liner and have your answer in a few moments...
neyme 5y ago
100%. The other day I wanted to do some transformations on a file using the like of regex, sed and grep. After 1 hour I just gave up and thought "hey, why not try doing it in Python". It took me just 5 minutes to write a small script, which was eye opening. Since then, I just write python scripts instead of toiling with commands I don't know.
bityard 5y ago
> I really dislike sed, awk, and all these utilities because I just can’t remember how to use them. Every time I want to use one of these tools I have to relearn the stuff, and then after a while I forget how to use it. If I don’t use it all the time I just forget. I'm going to let you in on a little secret: many of us who have been doing development or systems administration in some form another for multiple decades are the same way. I have a terrible memory, but it turns out that does not have to be a big hindrance for tech work. When I'm working (or tinkering at home), I keep a tab to my personal wiki open at all times. I will not usually bother to write something down the first time I do it. But I have to google it twice, I throw it into my wiki. This does two things: 1) It makes me (slightly) more likely to actually remember it for next time, and 2) The next time I need the information, I know where I can find it without having to wade through SEO-encumbered blog spam or outdated StackOverbutt answers. A database of personal notes is a powerful multiplier for technical ability and productivity. To the point that whenever I interview a candidate for a job, one of the things I ask is what system they keep their notes in. Their system doesn't matter at all to me but if the answer is none, that's a definite strike against. >I really don’t think non-interactive CLIs are a good way to do complicated tasks. Many of us thrive on the Unix command line because of the raw power you can wield with it. "Why waste time trying to learn arcane shell commands when you can just write a script in Python?" is a common refrain I hear from developers with only a few years under their belt. My response to that is, "Why would I waste time writing a Python script when I can do it in one line of shell?"
- rustyminnow 5y ago
  I 100% agree: If I have to look something up twice, it's going in my wiki. The first time you look something up, it might not be obvious if you'll ever use that info again, so why waste the time and clog up your notes with a bunch of noise? But if you've had to look something up twice, odds are good you'll have to look it up again someday. And like you said, writing a note in a way that's clear and understandable helps you to recall the thing a little better next time
- nomel 5y ago
  > personal wiki open at all times Any suggestion for a remotely accessible personal wiki? (I'm assuming that access is restricted for potential "personal" stuffs).
csydas 5y ago
This is everyone :) I love awk and whenever I __do__ find a task where awk helps me, it's really an amazing experience and it is always a boon, never a burden. But I really suck at remembering the syntax as I only hit tasks that awk solves well rarely. What I do remember is situations where awk definitely can likely help. Same with sed, tr, etc. Keep in mind, every time we automate something or introduce a new tool/process, it's a cost benefit analysis; when I have dozens of gigs of logs to parse to figure out which hosts out of hundreds are having issues, it's a no-brainer for me; the 30 minutes to revisit a few awk and bash tricks is a far better investment than trying to grep/scroll my way through literally millions of lines of logs/code. When I first learned awk I definitely went through a 'hammer' phase (i.e., when you have a hammer, everything looks like a nail); once phase passed, I find that I'm much more disciplined on when I break out awk or similar tools. Ultimately it's about identifying when you have the "right" tool for a specific job; I might need to invest time to revisit some syntax, but if the end result is that my 30 minutes saves hours, then it's a good 30 minutes. I'm at the stage where the basic text munging with awk is a no-brainer (or at least I remember the mistakes I made previously and how I fixed them); this was achievable only after a few projects where I did have to spend a bit more time wrangling awk than I preferred, but it saved me a lot of time in the future. Just knowing what I __can__ do with a given tool helps me make educated decisions on where I should invest my time on a given issue.

by baby 5y ago

Debug_Overload 5y ago

How long does the relearning take exactly? My experience has been that if you've used these tools before, it's pretty easy to remember or refresh your memory. In that sense, they're not very different from other tools/syntaxes that you haven't used in a while. And if you do use them frequently enough, you might not even need to refresh/relearn anything.

baby 5y ago
I basically gave up on relearning them considering the number of times I’ve had to do it and the amount of effort involved to relearn them. Maybe it’s just me, but I just write python now.
- sam_lowry_ 5y ago
  You need to have a mental model for each tool and some idea of syntax associated with it. Once I learned about IFS,OFS, BEGIN, END and {}, as well as the fact that variables and their references have the same syntax, unlike in Perl... I could pick up coding in awk at any moment
  - chasil 5y ago
    IFS is the POSIX shell; in awk, it's just FS. I think by {} you mean an argument to the find command.
    - sam_lowry_ 5y ago
      Indeed. Thanks for the hint.
  - kbenson 5y ago
    Perl has BEGIN and END as well, and when using it with -n or -p for line processing mode uses -F to set word boundaries just like awk. In fact, it shared many flags and features with awk, which it was meant to be able to be used as a more powerful version of the same tool.
- ivanhoe 5y ago
  Makes sense if python is what you otherwise use for everyday work, it's always easiest if you can avoid context switching. To me it's bash, sed and awk for the basic text-processing things, and then perl for anything that requires more programming. These are very similar syntaxes (and ways of thinking I'd say), so they fit together great - and also are close enough to php and js that I do at work to make the transition painless. But if I had to write it in python or ruby or God forbid C++ - which all I used a lot, but long time ago - I'm sure it would be a big struggle for me now.
- chasil 5y ago
  The full specification for the AWK language is 40 pages in the book written by the authors. I sometimes forget the exact syntax for pipes and file open/close, but it's not hard to find in 40 pages. AWK is in busybox, and a lot of other places that Python simply cannot go. The control structure syntax is also the same a C/JavaScript/PHP/C++ etc., so it certainly does not hurt to know it. https://archive.org/download/pdfy-MgN0H1joIoDVoIC7/The_AWK_Programming_Language.pdf https://archive.org/download/pdfy-MgN0H1joIoDVoIC7/The_AWK_P...

thaumaturgy 5y ago

I'm mostly with you (except for your last sentence). I've picked up a lot of languages over the years, and can fall back into most of them pretty fast, but sed and awk especially are hard. On the occasions that they really are the best tool for the job, I get actual books out and set aside a window of a few hours to plod through it. I think the problem is that they fall into an uncanny valley between "simple tool" and "verbose programming language"; they are really terse languages wearing a simple tool's sheepskin.

sitkack 5y ago
If I measure time to solve a problem and not line count, using a language like Python or Ruby might be 2x the lines , for a line count approaching zero anyways, not sure that matters. Performance is comparable and I don't need to relearn anything and someone else who doesn't know awk/sed doesn't need to learn it. And when one is using awk/sed that script is often invoked from a shell, bash for instance. And since the awk/sed can't hold its own as a general purpose programming language, now I am writing bash code, a personal sin. Awk/sed made sense for the time and place they are from, but things have changed.
- thaumaturgy 5y ago
  Yes, this is a Standard HN Reply: the "Specialist's Rebuttal". It's usually in some form of, "___ is obsolete, ___ is better in every way, everyone should just be using ___ now." Okay, acknowledged. Python and Ruby are superior in every regard. Would you mind letting us chat about sed and awk now for a minute?
  - yesenadam 5y ago
    I replied to someone's "There is no reason for anyone to program in C nowadays" recently on here with "But I like programming in C". They hadn't considered that reason in their mental survey of Every Possible Reason. Also.. I use Awk every day, for all kinds of things. I forget the order of parameters in split, sub etc, which takes a few seconds to look up with man awk. I got into sed a while ago but never use it. Awk, together with sort and uniq sometimes, is all I need. And it is interactive. You have an input. Print that out and look what needs to be done. Write the first transforming step in Awk. See what that did. Add the next step. See what that did. Extremely interactive. After a few minutes I have a one-liner (often 2 or 3 wrapped lines) that does exactly what I want, and easy to adapt to other similar things. I have very large bash history enabled so they stay Ctrl-R-accessible.

aidos 5y ago

That’s just because you don’t use them often though. If they were a daily driver it’d be different. I think that if you have a good mental model of how awk works, then you know times that it’s the right tool for the job. And on those occasions you’ve got plenty of time to go search for some syntax, make a coffee, maybe even go for a run all with change to spare compared to using pretty much anything else. I will admit, that if I can do it with a few greps, seds and cuts then I tend to just use those first, because there no thinking involved for me.

ZeroGravitas 5y ago

I mostly find them useful when you can't be sure anything else will be installed. What we really need is something that can "compile" down to sed for deployment in that kind of environment. Then that compiler can contain all the weird stuff you need to remember.

s1291 5y ago

cheat.sh is your friend. E.g for sed: curl cheat.sh/sed Will show you several examples.

News-Dog 5y ago
cheat.sh & cht.sh < Nice! https://github.com/chubin/cheat.sh https://github.com/chubin/cheat.sh Also, LDP: Advanced Bash-Scripting Guide: https://tldp.org/guides.html https://tldp.org/guides.html

musicale 5y ago

Since sed is the stream version of ed/vi, it used to be easier to use and remember for people who used ed/vi as their primary editor.

mellavora 5y ago

awk is always a struggle for me to re-learn, but sed is simple. I basically use it for simple text substitutions; in that case it is the same as what I use for vim editing.

adrianscheff 5y ago

Haha, one of the reasons I wrote this. It's also a way of practicing and reminding myself sed.

OJFord 5y ago

Personally I'm fine with using a complicated expression (or a pipeline) built out of basic parts where it's not necessary (because there's some whizzbang operator that'll make it so much more succinct) if that's what I can remember or what is clearer to understand with fresh eyes.

superasn 5y ago

Which is why I've always stuck to `perl -pi -e`. It may not be as powerful but it's just one thing to learn. Also it understands regexs like '\d', etc which are kinda universal but for some reason don't work with many of these utils.

mdaniel 5y ago
I also read once that perl was the most ubiquitous tool across all Unixen flavors, but I don't have a link to cite for it
gnubison 5y ago
Perl is certainly more powerful than sed or awk :)
kbenson 5y ago
I wouldn't call having the de-facto standard for an advanced regular expression system, a full programming language and the entirety of the CPAN module system easily pulled in and used less powerful. I use sed for a lot of simple stuff, but I use Perl all the time for more advanced stuff. Perl is great for grabbing complex chunks out of a line and doing something with them that requires a bit of state. For example, writing a quick and dirty logfile parser to figure out how many requests and how many bytes were used by each IP over a time period (the time period limiting usually being a prior grep before being passed to Perl). I've probably taken 60 seconds to write out something to parse exactly what I want from a log hundreds of times.

zouhair 5y ago

In these instances where you find a one liner useful but you use it rarely, just put it in a function with comments on how it works. When you want to use it the function is ready and the explanation too.

memetomancer 5y ago

May I ask what is the alternative? Creating an entire application for a single use? designing a generalized case and making a monolithic application with lots of obscure options? What about setting up the project for the application, setting up instrumentation and testing, debugging, etc.? Even supposing that all that extra work makes sense for your unique string manipulation needs, what about the next time something like this comes up but is out of scope for your first tool? Do you end up with lots of little applications, with lots of little features and arguments? When you come across that similar-but-not-quite need down the road, do you have to scour your library of full blown apps to find something that fits? Not to mention maintaining them against future system updates... OR - you could just force yourself to get over the hump, learn the conceptual basics at play and then when you come across a novel situation down the line you can just rattle off a tailored one-liner and have your answer in a few moments...

neyme 5y ago

100%. The other day I wanted to do some transformations on a file using the like of regex, sed and grep. After 1 hour I just gave up and thought "hey, why not try doing it in Python". It took me just 5 minutes to write a small script, which was eye opening. Since then, I just write python scripts instead of toiling with commands I don't know.

bityard 5y ago

> I really dislike sed, awk, and all these utilities because I just can’t remember how to use them. Every time I want to use one of these tools I have to relearn the stuff, and then after a while I forget how to use it. If I don’t use it all the time I just forget. I'm going to let you in on a little secret: many of us who have been doing development or systems administration in some form another for multiple decades are the same way. I have a terrible memory, but it turns out that does not have to be a big hindrance for tech work. When I'm working (or tinkering at home), I keep a tab to my personal wiki open at all times. I will not usually bother to write something down the first time I do it. But I have to google it twice, I throw it into my wiki. This does two things: 1) It makes me (slightly) more likely to actually remember it for next time, and 2) The next time I need the information, I know where I can find it without having to wade through SEO-encumbered blog spam or outdated StackOverbutt answers. A database of personal notes is a powerful multiplier for technical ability and productivity. To the point that whenever I interview a candidate for a job, one of the things I ask is what system they keep their notes in. Their system doesn't matter at all to me but if the answer is none, that's a definite strike against. >I really don’t think non-interactive CLIs are a good way to do complicated tasks. Many of us thrive on the Unix command line because of the raw power you can wield with it. "Why waste time trying to learn arcane shell commands when you can just write a script in Python?" is a common refrain I hear from developers with only a few years under their belt. My response to that is, "Why would I waste time writing a Python script when I can do it in one line of shell?"

rustyminnow 5y ago
I 100% agree: If I have to look something up twice, it's going in my wiki. The first time you look something up, it might not be obvious if you'll ever use that info again, so why waste the time and clog up your notes with a bunch of noise? But if you've had to look something up twice, odds are good you'll have to look it up again someday. And like you said, writing a note in a way that's clear and understandable helps you to recall the thing a little better next time
nomel 5y ago
> personal wiki open at all times Any suggestion for a remotely accessible personal wiki? (I'm assuming that access is restricted for potential "personal" stuffs).

csydas 5y ago

This is everyone :) I love awk and whenever I __do__ find a task where awk helps me, it's really an amazing experience and it is always a boon, never a burden. But I really suck at remembering the syntax as I only hit tasks that awk solves well rarely. What I do remember is situations where awk definitely can likely help. Same with sed, tr, etc. Keep in mind, every time we automate something or introduce a new tool/process, it's a cost benefit analysis; when I have dozens of gigs of logs to parse to figure out which hosts out of hundreds are having issues, it's a no-brainer for me; the 30 minutes to revisit a few awk and bash tricks is a far better investment than trying to grep/scroll my way through literally millions of lines of logs/code. When I first learned awk I definitely went through a 'hammer' phase (i.e., when you have a hammer, everything looks like a nail); once phase passed, I find that I'm much more disciplined on when I break out awk or similar tools. Ultimately it's about identifying when you have the "right" tool for a specific job; I might need to invest time to revisit some syntax, but if the end result is that my 30 minutes saves hours, then it's a good 30 minutes. I'm at the stage where the basic text munging with awk is a no-brainer (or at least I remember the mistakes I made previously and how I fixed them); this was achievable only after a few projects where I did have to spend a bit more time wrangling awk than I preferred, but it saved me a lot of time in the future. Just knowing what I __can__ do with a given tool helps me make educated decisions on where I should invest my time on a given issue.