13 ms·

Useful sed scripts and patterns

528 points by adrianscheff 5y ago

tyingq 5y ago
"keep the first word of every line (where word is defined by alnum chars + underscores for simplicity sake)" sed -E s_[a-zA-Z0-9_]+.*_\1_' file.txt You're missing an opening single quote and a set of parentheses for this to work. It feels like you should have a unit test setup for these snippets, given that there are other typos found in other comments here.
RNCTX 5y ago
wrote this ages ago, if you want something to put all your office docs and pdfs in a solr database on FreeBSD (indexable by page number), knock yourself out... https://gist.github.com/awhileback/1fffac899d9321a6a9ec15bc9ec58e29 https://gist.github.com/awhileback/1fffac899d9321a6a9ec15bc9... Requires some ocr / pdf parsing / metadata tools, see the comments.
xrd 5y ago
When I see anything that leaves a backup file as .bak with inline replacing, it makes me reminisce over Perl. Perl one liners are still hard to beat.
neeeeees 5y ago
“s '1,$ s/foo/bar/' file.txt”. should be replacement from 1 to end, not 5 to end (#4 on your list)
- jachee 5y ago
  I just submitted a PR to fix that. :)
  - jachee 5y ago
    …and I closed it as a dupe already. I was seconds slower than the other contributor. :)
silisili 5y ago
Sed is probably the one tool I owe my career most to. I made a living mostly on it for a few years as an ops guy, and I didn't even know half of these use cases. The variable delimiter alone was such a beautifully thought out decision that made everything easier, that I likely would have never considered had I written it.
- adrianscheff 5y ago
  I'm glad to hear that. I'm learning (and re-learning) sed myself, specially after long periods of not using it. So in a way I've written it for me too. Btw, there is one more use case of a variable delimiter which is more arcane (and you can combine it with the other custom delimiter) `sed -s '\_/bin/bash_s:grep:egrep:' myfile.txt`
snidane 5y ago
Remove csv header. sed 1d Select line 123 sed -n 123p # or sed -n 123{p;q}
- dolmen 5y ago
  sed -n '123{p;q;}'
rackjack 5y ago
sed actually provides a scripting language, and s/foo/bar is just statement in that language. This surprised me when I first learned about it.
Waterluvian 5y ago
Something I crave in the worst way is a sensible story for accumulating a “toolbox” of snippets and scripts and commands. They don’t need to be ready to go, but ideally: - natural language searchable - add a small description - CLI to search, examine, and copy - not a <favourite text editor> plugin - sync with a GitHub repo.
- f0e4c2f7 5y ago
  One of the best hackers I ever worked with turned me onto this method. Use a flat text in the editor of your choice. Search as needed. Tags work because you can seach them. You delimit each section as a note. I do it like this --- Next note --- It seems limited and primative, you don't get any markup or fun stuff. But it's soooo good. There is nothing to break or update. Seach is the method I use most anyway in my other note taking apps. And you wind up putting everything in there, including commands that you're staging to run. Those staged commands become snippets later.
  - imiric 5y ago
    I keep a commands.org file for exactly this purpose. Whenever a command is useful to maintain outside of my shell history, it gets documented and placed there, along with variations, things to look out for, etc. Eventually I'd like to publish it as a microblog for such snippets ala commandlinefu.com, but priorities and laziness get in the way. :/
  - notatoad 5y ago
    Text files are good, but even more primitive and also awesome: I find a sheet of paper taped to the wall in my office is really underrated. Being able to just look up and see it is awesome, as well as the natural limitation to only as much stuff as will fit on a sheet of paper. Every now and then I start a new sheet, dropping off the stuff I've memorized.
  - sen 5y ago
    I still use flat text files for 90% of my daily note taking. Have done since the early 90s, and the same notes I took then are still working in the same folder I use today. I sync it to my various devices (every device and platform has a text editor), and it’s never failed me. I have txt files for commands I want to remember for each OS I daily use (linuxcmds.txt, windowscmds.txt, maccmds.txt) and split them the same way you said but 5 dashes. Also keep txt files for install/setup notes for each os, links to every program I want to install in fresh installs, etc etc. Text files work on every single OS/device, the format is stable, it’s easily searchable, easily shareable, and my entire 30 years of notes is measured in megabytes.
saurik 5y ago
There are a ton of little errors in this post (at the time of my leaving this comment, of course): the variable delimiter example is missing the close _, the first word example is missing the open ', and the except line 5 example is missing a space; meanwhile, these snippets will cause bad habits, such as thinking about "words" as "things involving letters", using -r instead of -E, and a massive knowledge gap with respect to appropriate use of semicolons... I'm also really confused as to why sed is suddenly "s" in one of the examples. This just isn't a very good reference, and I'm not sure if the author thinks about how "c" works, but it really needs a \ afterwards, no matter what the one specific version of sed you are using might allow :/. As someone who loves sed so much he nigh unto regrets how much he uses it: "beware". FWIW, I'd argue that the path to true sed mastery eventually goes through the hold space, which isn't mentioned here. For some real fun, and an exercise that might change how you mentally model what sed is capable of, check out SedSokoban. https://aurelio.net/projects/sedsokoban/ https://aurelio.net/projects/sedsokoban/
- imiric 5y ago
  While this might not be the ultimate sed reference, it turned me onto some patterns I haven't used before. The examples aren't meant to be used directly and will involve some trial and error before they're useful, but knowing what's possible makes it a very valuable reference.
- adrianscheff 5y ago
  Thank you for the feedback and the constructive criticism. Regarding the hold space I found it very confusing to use and understand. IMO it represents the 80/20 of sed (80% effort for 20% results). You're right, I'm using -r even when it's not necessary. To my defense I think it's a good habit to have since without it regex expressions are painful to write. I didn't considered that using -E it's a better choice but I'll correct that now. (one might argue again that typing -r is easier than -E :D ). Regarding the definition of words - I also thought of that when I wrote that snippet. I know it's not the complete regex for a word and that word regex patterns might differ. And I was probably a bit lazy - but I'll correct it presently. I'd also like to say that I didn't write this as an absolute and ultimate reference. If I'm honest I wrote this as much to teach others as to solidify this knowledge myself. Now since it seems it gained traction I'm kinda obligated to make this better, no? Darn. :) PS: if you'd like to help me make this better please submit a pull request or leave a comment here. Looking at your profile I see that you try to limit your online time so I probably shouldn've asked. :P
atsaloli 5y ago
See also https://catonmat.net/sed-one-liners-explained-part-one https://catonmat.net/sed-one-liners-explained-part-one
mgerdts 5y ago
Instead of wrapping sed with a bash script to get multiple expressions to do their things, I normally use 'sed -e'. sed -e s/a/A/ -e s/foo/BAR/ -e s/hello/HELLO/ I use -e often enough that I usually use -e whether I expect to use more than one expression or not. There's a reasonable chance that I will be going back into my history and will need to add another, which is easier if the first -e is already there. Also, 'sed -e expr1 -e expr2' gives the same results as 'sed expr1 | sed expr2'. In both cases, the order of the expressions matter: later expressions may change things altered by the first. $ echo foo | sed -e 's/f/g/' -e 's/goo/go/' go
- ephaeton 5y ago
  Then there's sed -e 'cmd;cmd;cmd' as well.
asicsp 5y ago
Nice descriptions, but some of them need changes to match the command. For example: sed '1~2p' file.txt #needs -n option s '1,$ s/foo/bar/' file.txt #s --> sed and 1,$ --> 5,$ Many commands using `-r` do not need the option for the command used (for ex: `sed -r '/start/q'`). Also, using `-E` is preferred instead of `-r` since some of the other implementations support this option but not `-r`. --- I wrote a book on GNU sed with plenty of examples and exercises: https://github.com/learnbyexample/learn_gnused https://github.com/learnbyexample/learn_gnused It is free to read online and there's a detailed chapter for learning BRE/ERE regex flavor as well.
- adrianscheff 5y ago
  Thank you for the constructive feedback, I've corrected the mistakes pointed out. I've also changed -r to -E based on your (and some other people's) advice.
baby 5y ago
I really dislike sed, awk, and all these utilities because I just can’t remember how to use them. Every time I want to use one of these tools I have to relearn the stuff, and then after a while I forget how to use it. If I don’t use it all the time I just forget. I really don’t think non-interactive CLIs are a good way to do complicated tasks.
- Debug_Overload 5y ago
  How long does the relearning take exactly? My experience has been that if you've used these tools before, it's pretty easy to remember or refresh your memory. In that sense, they're not very different from other tools/syntaxes that you haven't used in a while. And if you do use them frequently enough, you might not even need to refresh/relearn anything.
  - baby 5y ago
    I basically gave up on relearning them considering the number of times I’ve had to do it and the amount of effort involved to relearn them. Maybe it’s just me, but I just write python now.
    - sam_lowry_ 5y ago
      You need to have a mental model for each tool and some idea of syntax associated with it. Once I learned about IFS,OFS, BEGIN, END and {}, as well as the fact that variables and their references have the same syntax, unlike in Perl... I could pick up coding in awk at any moment
      - chasil 5y ago
        IFS is the POSIX shell; in awk, it's just FS. I think by {} you mean an argument to the find command.
        sam_lowry_ 5y ago
        Indeed. Thanks for the hint.
vfinn 5y ago
Waiting for an AI powered shell, or something similar, so that one can avoid wasting time on these trivial tasks that require too much attention if solved with traditional methods: "I made an OpenAI-powered Linux shell that guesses your bash command": https://www.youtube.com/watch?v=j0UnS3jHhAA https://www.youtube.com/watch?v=j0UnS3jHhAA
undershirt 5y ago
Is there a way we ask these kinds of question in plain english? and then have it return a snippet like this
- throwamon 5y ago
  Sounds exactly like GitHub Copilot.
- teddyh 5y ago
  There were efforts to do that for general programming; traces of those efforts survive today as COBOL and SQL.
- drran 5y ago
  StackOverflow
  - SAI_Peregrinus 5y ago
    StackOverflow: "There are no stupid questions, but there are a lot of inquisitive idiots. Closing as duplicate of <unrelated question>."
blunte 5y ago
Mastering one tool is probably wise, but it's also good to know which tools may be better suited for some purposes. For "keep the first word of every line", I prefer awk: awk '{ print $1 }' < file.txt
- brabel 5y ago
  I agree with the people here saying that it's better to just use your favourite scripting language unless your job is mostly about writing little bash scripts like that (maybe sysops people?)... Here's a groovy script to do the same: new File('file.txt').eachLine { println it.split()[0] } Not as neat as awk, but not much worse either... and as I use Groovy a lot for testing, and its syntax is simplified Java (which I use a lot as well) it's just much better for me... if I had to ssh into other servers where Groovy or Lisp (my other favourite "scripting" language) are not available, then yeah, I would learn awk better (I actually like awk, and use it sometimes for the rare occasion I don't have groovy/lisp installed).
  - blunte 5y ago
    In cases where it's a one time use, anything that gets you an answer easily is great. Could be Groovy, Ruby, Python, Perl, etc. or of course these bin tools. But if it's going to be used a lot, and start-up time matters, then some of the full programming languages can become expensive for the temporary load they create just spinning up. In those cases, the standard Unix tools tend to be so much faster and lighter that it's worth learning them.
    - anthk 5y ago
      Perl is much better than Pyhon for that for obvious reasons, Perl is like an improved sed with stereoids with some C inspiration.
  - dolmen 5y ago
    Use the best tool for the job. sed is just a DSL (domain specific language) specifically designed for for manipulating text files line by line. Note that for matching strings even in Groovy/AWK/Python/Perl/... you'll probably use regexp which are another DSL. Would you recommend to use string functions instead?
ReleaseCandidat 5y ago
You can avoid typing errors in commands or code as part of Markdown files (or any other markup language) by using some form of 'literate programming' like Jupyter notebooks. The default IPython kernel supports shell by putting a '!' before each line of shell commands. You can get Jupyter kernels for almost any programming language - though many work on Linux only. Or you can use Org-Mode (with babel), if you're an Emacs user. That's how I write all documentation involving some sort of code.
drKarl 5y ago
Have you submitted them to commandlinefu.com?
WastingMyTime89 5y ago
I understand the usefulness of this kind of command listing for people who just use sed occasionnaly and want to quickly copy/paste something still I think just learning how sed works is actually easier than treating it like magic. It's not awk. Sed is both logical and simple. Sed is the silent version of ed. It's a full line-oriented editor. You give it commands and it executes them on lines, one line at a time. Commands always have the same form: [lines range] [action][parameters] You just have to learn how ranges are defined and what the actions are then you can check the documentation for what the parameters are when you need to and you will be good to go forever. As a bonus, once you realize there is more actions than just s for substitute, you start realizing that people sometimes do very convoluted things with sed because they don't know how to use the other actions. For example, there are actions to join lines (j), to delete lines (d), to prefilter lines based on a regex (g and G).
- twic 5y ago
  > I think just learning how sed works is actually easier than treating it like magic. It's not awk. As someone with over a decade of heavy sed use, who only got into awk recently, i'd say awk is far easier and less magical. Awk is just another programming language, for the most part. Using sed for anything substantial requires you to think in strange directions, to find ways to thread state and control flow through the tiny holes the language gives you.
  - WastingMyTime89 5y ago
    I think I both agree and disagree. As you rightfully pointed, I wouldn’t do anything overly complicated with sed because it really is a line editor and anything which is not line based and can’t be explained simply in the context of using an editor will quickly seem clunky and overly complicated. I wouldn’t do anything period with awk. I agree with you that it’s just another programming language but one which feels extremely dated with both a terrible syntax and awkward semantics. The rare times I have encountered something done with awk it was either so simple it should have been done with sed or so complicated I wish a proper scripting language like python or perl would have been used instead. As far as I’m concerned, awk is a tool without a proper use-case. It’s always better to use something else.
visualphoenix 5y ago
This is awkward… I know it’s hip and cool to ask for donations, but for a tiny collection of sed scripts which aren’t including detailed explanations? I feel like sed is what deserves the donation… http://sed.sourceforge.net/sed1line.txt http://sed.sourceforge.net/sed1line.txt ? https://www.fsf.org/about/ways-to-donate/ https://www.fsf.org/about/ways-to-donate/
- adrianscheff 5y ago
  Hey there! Thank you for the input! I'm sorry you feel that asking for donations is unwarranted. I agree wholeheartedly that FSF deserves all the love it can get. I'm not asking to get rich or take the spotlight - I'm asking so I can keep improving that guide & create more. I didn't added explanations (although I wanted too) since this was intended as a quick tips page for sed. I think there are much better guides than I could ever make (including the info page). In a way it works better since it's more digestible. Thank you for the link you provided - it looks awesome! I'll look into it and append the existing guide if needed (while giving credits of course)
krinchan 5y ago
I use sed -ibak ‘10d’ ~/.ssh/.known_hosts so many times a week. awk, sed, and regex really are the two programs I am happy to have learned well at my first job. They’ve helped me immensely throughout my 10 years as someone who bridges developer and ops roles.
- mdaniel 5y ago
  For that one specifically, you may want to check your copy of "ssh-keygen" as it has recently(?) learned to manage those known_hosts entries without resorting to text file manipulation (`ssh-keygen -r something.example.com` for example)
cjvirtucio 5y ago
I'm annoyed at how sed has --regexp-extended, while grep has `--extended-regexp`. I'm sure there's a reason for it, but it always makes me stop and think which command has which flag.
jaimehrubiks 5y ago
What is the easiest way to make sure a "field, or option" is present in a file. Similar to what ansible's lineinfile does? For example suppose I need to add this setting to /etc/ssh/sshd_config: PermitRootLogin yes One idea that comes to mind is to first delete the line with the line shown in this post, and then add it back: Regex sed -E '/^#/d' file.txt - delete lines where regex matches sed -E '/^PermiteRootLogin/d' /etc/ssh/sshd_config ; echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config Is there a more straightforward way with sed?
- pnutjam 5y ago
  https://stackoverflow.com/questions/13014055/sed-command-how-to-replace-if-exists-else-just-insert https://stackoverflow.com/questions/13014055/sed-command-how...
jagged-chisel 5y ago
> Print one line > > sed -n '10p' myfile.txt Which line? Why p? I can make assumptions that this means “line 10, print” but easy-to-explain could go a small step further and complete the explanation.
max_hammer 5y ago
https://edoras.sdsu.edu/doc/sed-oneliners.html https://edoras.sdsu.edu/doc/sed-oneliners.html

528 points by adrianscheff 5y ago

"keep the first word of every line (where word is defined by alnum chars + underscores for simplicity sake)" sed -E s_[a-zA-Z0-9_]+.*_\1_' file.txt You're missing an opening single quote and a set of parentheses for this to work. It feels like you should have a unit test setup for these snippets, given that there are other typos found in other comments here.

RNCTX 5y ago

wrote this ages ago, if you want something to put all your office docs and pdfs in a solr database on FreeBSD (indexable by page number), knock yourself out... https://gist.github.com/awhileback/1fffac899d9321a6a9ec15bc9ec58e29 https://gist.github.com/awhileback/1fffac899d9321a6a9ec15bc9... Requires some ocr / pdf parsing / metadata tools, see the comments.

xrd 5y ago

When I see anything that leaves a backup file as .bak with inline replacing, it makes me reminisce over Perl. Perl one liners are still hard to beat.

neeeeees 5y ago

“s '1,$ s/foo/bar/' file.txt”. should be replacement from 1 to end, not 5 to end (#4 on your list)

jachee 5y ago
I just submitted a PR to fix that. :)
- jachee 5y ago
  …and I closed it as a dupe already. I was seconds slower than the other contributor. :)

silisili 5y ago

Sed is probably the one tool I owe my career most to. I made a living mostly on it for a few years as an ops guy, and I didn't even know half of these use cases. The variable delimiter alone was such a beautifully thought out decision that made everything easier, that I likely would have never considered had I written it.

adrianscheff 5y ago
I'm glad to hear that. I'm learning (and re-learning) sed myself, specially after long periods of not using it. So in a way I've written it for me too. Btw, there is one more use case of a variable delimiter which is more arcane (and you can combine it with the other custom delimiter) `sed -s '\_/bin/bash_s:grep:egrep:' myfile.txt`

snidane 5y ago

Remove csv header. sed 1d Select line 123 sed -n 123p # or sed -n 123{p;q}

dolmen 5y ago
sed -n '123{p;q;}'

rackjack 5y ago

sed actually provides a scripting language, and s/foo/bar is just statement in that language. This surprised me when I first learned about it.

Waterluvian 5y ago

Something I crave in the worst way is a sensible story for accumulating a “toolbox” of snippets and scripts and commands. They don’t need to be ready to go, but ideally: - natural language searchable - add a small description - CLI to search, examine, and copy - not a <favourite text editor> plugin - sync with a GitHub repo.

f0e4c2f7 5y ago
One of the best hackers I ever worked with turned me onto this method. Use a flat text in the editor of your choice. Search as needed. Tags work because you can seach them. You delimit each section as a note. I do it like this --- Next note --- It seems limited and primative, you don't get any markup or fun stuff. But it's soooo good. There is nothing to break or update. Seach is the method I use most anyway in my other note taking apps. And you wind up putting everything in there, including commands that you're staging to run. Those staged commands become snippets later.
- imiric 5y ago
  I keep a commands.org file for exactly this purpose. Whenever a command is useful to maintain outside of my shell history, it gets documented and placed there, along with variations, things to look out for, etc. Eventually I'd like to publish it as a microblog for such snippets ala commandlinefu.com, but priorities and laziness get in the way. :/
- notatoad 5y ago
  Text files are good, but even more primitive and also awesome: I find a sheet of paper taped to the wall in my office is really underrated. Being able to just look up and see it is awesome, as well as the natural limitation to only as much stuff as will fit on a sheet of paper. Every now and then I start a new sheet, dropping off the stuff I've memorized.
- sen 5y ago
  I still use flat text files for 90% of my daily note taking. Have done since the early 90s, and the same notes I took then are still working in the same folder I use today. I sync it to my various devices (every device and platform has a text editor), and it’s never failed me. I have txt files for commands I want to remember for each OS I daily use (linuxcmds.txt, windowscmds.txt, maccmds.txt) and split them the same way you said but 5 dashes. Also keep txt files for install/setup notes for each os, links to every program I want to install in fresh installs, etc etc. Text files work on every single OS/device, the format is stable, it’s easily searchable, easily shareable, and my entire 30 years of notes is measured in megabytes.

saurik 5y ago

There are a ton of little errors in this post (at the time of my leaving this comment, of course): the variable delimiter example is missing the close _, the first word example is missing the open ', and the except line 5 example is missing a space; meanwhile, these snippets will cause bad habits, such as thinking about "words" as "things involving letters", using -r instead of -E, and a massive knowledge gap with respect to appropriate use of semicolons... I'm also really confused as to why sed is suddenly "s" in one of the examples. This just isn't a very good reference, and I'm not sure if the author thinks about how "c" works, but it really needs a \ afterwards, no matter what the one specific version of sed you are using might allow :/. As someone who loves sed so much he nigh unto regrets how much he uses it: "beware". FWIW, I'd argue that the path to true sed mastery eventually goes through the hold space, which isn't mentioned here. For some real fun, and an exercise that might change how you mentally model what sed is capable of, check out SedSokoban. https://aurelio.net/projects/sedsokoban/ https://aurelio.net/projects/sedsokoban/

imiric 5y ago
While this might not be the ultimate sed reference, it turned me onto some patterns I haven't used before. The examples aren't meant to be used directly and will involve some trial and error before they're useful, but knowing what's possible makes it a very valuable reference.
adrianscheff 5y ago
Thank you for the feedback and the constructive criticism. Regarding the hold space I found it very confusing to use and understand. IMO it represents the 80/20 of sed (80% effort for 20% results). You're right, I'm using -r even when it's not necessary. To my defense I think it's a good habit to have since without it regex expressions are painful to write. I didn't considered that using -E it's a better choice but I'll correct that now. (one might argue again that typing -r is easier than -E :D ). Regarding the definition of words - I also thought of that when I wrote that snippet. I know it's not the complete regex for a word and that word regex patterns might differ. And I was probably a bit lazy - but I'll correct it presently. I'd also like to say that I didn't write this as an absolute and ultimate reference. If I'm honest I wrote this as much to teach others as to solidify this knowledge myself. Now since it seems it gained traction I'm kinda obligated to make this better, no? Darn. :) PS: if you'd like to help me make this better please submit a pull request or leave a comment here. Looking at your profile I see that you try to limit your online time so I probably shouldn've asked. :P

atsaloli 5y ago

See also https://catonmat.net/sed-one-liners-explained-part-one https://catonmat.net/sed-one-liners-explained-part-one

mgerdts 5y ago

Instead of wrapping sed with a bash script to get multiple expressions to do their things, I normally use 'sed -e'. sed -e s/a/A/ -e s/foo/BAR/ -e s/hello/HELLO/ I use -e often enough that I usually use -e whether I expect to use more than one expression or not. There's a reasonable chance that I will be going back into my history and will need to add another, which is easier if the first -e is already there. Also, 'sed -e expr1 -e expr2' gives the same results as 'sed expr1 | sed expr2'. In both cases, the order of the expressions matter: later expressions may change things altered by the first. $ echo foo | sed -e 's/f/g/' -e 's/goo/go/' go

ephaeton 5y ago
Then there's sed -e 'cmd;cmd;cmd' as well.

asicsp 5y ago

Nice descriptions, but some of them need changes to match the command. For example: sed '1~2p' file.txt #needs -n option s '1,$ s/foo/bar/' file.txt #s --> sed and 1,$ --> 5,$ Many commands using `-r` do not need the option for the command used (for ex: `sed -r '/start/q'`). Also, using `-E` is preferred instead of `-r` since some of the other implementations support this option but not `-r`. --- I wrote a book on GNU sed with plenty of examples and exercises: https://github.com/learnbyexample/learn_gnused https://github.com/learnbyexample/learn_gnused It is free to read online and there's a detailed chapter for learning BRE/ERE regex flavor as well.

adrianscheff 5y ago
Thank you for the constructive feedback, I've corrected the mistakes pointed out. I've also changed -r to -E based on your (and some other people's) advice.

baby 5y ago

I really dislike sed, awk, and all these utilities because I just can’t remember how to use them. Every time I want to use one of these tools I have to relearn the stuff, and then after a while I forget how to use it. If I don’t use it all the time I just forget. I really don’t think non-interactive CLIs are a good way to do complicated tasks.

Debug_Overload 5y ago
How long does the relearning take exactly? My experience has been that if you've used these tools before, it's pretty easy to remember or refresh your memory. In that sense, they're not very different from other tools/syntaxes that you haven't used in a while. And if you do use them frequently enough, you might not even need to refresh/relearn anything.
- baby 5y ago
  I basically gave up on relearning them considering the number of times I’ve had to do it and the amount of effort involved to relearn them. Maybe it’s just me, but I just write python now.
  - sam_lowry_ 5y ago
    You need to have a mental model for each tool and some idea of syntax associated with it. Once I learned about IFS,OFS, BEGIN, END and {}, as well as the fact that variables and their references have the same syntax, unlike in Perl... I could pick up coding in awk at any moment
    - chasil 5y ago
      IFS is the POSIX shell; in awk, it's just FS. I think by {} you mean an argument to the find command.
      - sam_lowry_ 5y ago
        Indeed. Thanks for the hint.

vfinn 5y ago

Waiting for an AI powered shell, or something similar, so that one can avoid wasting time on these trivial tasks that require too much attention if solved with traditional methods: "I made an OpenAI-powered Linux shell that guesses your bash command": https://www.youtube.com/watch?v=j0UnS3jHhAA https://www.youtube.com/watch?v=j0UnS3jHhAA

undershirt 5y ago

Is there a way we ask these kinds of question in plain english? and then have it return a snippet like this

throwamon 5y ago
Sounds exactly like GitHub Copilot.
teddyh 5y ago
There were efforts to do that for general programming; traces of those efforts survive today as COBOL and SQL.
drran 5y ago
StackOverflow
- SAI_Peregrinus 5y ago
  StackOverflow: "There are no stupid questions, but there are a lot of inquisitive idiots. Closing as duplicate of <unrelated question>."

blunte 5y ago

Mastering one tool is probably wise, but it's also good to know which tools may be better suited for some purposes. For "keep the first word of every line", I prefer awk: awk '{ print $1 }' < file.txt

brabel 5y ago
I agree with the people here saying that it's better to just use your favourite scripting language unless your job is mostly about writing little bash scripts like that (maybe sysops people?)... Here's a groovy script to do the same: new File('file.txt').eachLine { println it.split()[0] } Not as neat as awk, but not much worse either... and as I use Groovy a lot for testing, and its syntax is simplified Java (which I use a lot as well) it's just much better for me... if I had to ssh into other servers where Groovy or Lisp (my other favourite "scripting" language) are not available, then yeah, I would learn awk better (I actually like awk, and use it sometimes for the rare occasion I don't have groovy/lisp installed).
- blunte 5y ago
  In cases where it's a one time use, anything that gets you an answer easily is great. Could be Groovy, Ruby, Python, Perl, etc. or of course these bin tools. But if it's going to be used a lot, and start-up time matters, then some of the full programming languages can become expensive for the temporary load they create just spinning up. In those cases, the standard Unix tools tend to be so much faster and lighter that it's worth learning them.
  - anthk 5y ago
    Perl is much better than Pyhon for that for obvious reasons, Perl is like an improved sed with stereoids with some C inspiration.
- dolmen 5y ago
  Use the best tool for the job. sed is just a DSL (domain specific language) specifically designed for for manipulating text files line by line. Note that for matching strings even in Groovy/AWK/Python/Perl/... you'll probably use regexp which are another DSL. Would you recommend to use string functions instead?

ReleaseCandidat 5y ago

You can avoid typing errors in commands or code as part of Markdown files (or any other markup language) by using some form of 'literate programming' like Jupyter notebooks. The default IPython kernel supports shell by putting a '!' before each line of shell commands. You can get Jupyter kernels for almost any programming language - though many work on Linux only. Or you can use Org-Mode (with babel), if you're an Emacs user. That's how I write all documentation involving some sort of code.

drKarl 5y ago

Have you submitted them to commandlinefu.com?

WastingMyTime89 5y ago

I understand the usefulness of this kind of command listing for people who just use sed occasionnaly and want to quickly copy/paste something still I think just learning how sed works is actually easier than treating it like magic. It's not awk. Sed is both logical and simple. Sed is the silent version of ed. It's a full line-oriented editor. You give it commands and it executes them on lines, one line at a time. Commands always have the same form: [lines range] [action][parameters] You just have to learn how ranges are defined and what the actions are then you can check the documentation for what the parameters are when you need to and you will be good to go forever. As a bonus, once you realize there is more actions than just s for substitute, you start realizing that people sometimes do very convoluted things with sed because they don't know how to use the other actions. For example, there are actions to join lines (j), to delete lines (d), to prefilter lines based on a regex (g and G).

twic 5y ago
> I think just learning how sed works is actually easier than treating it like magic. It's not awk. As someone with over a decade of heavy sed use, who only got into awk recently, i'd say awk is far easier and less magical. Awk is just another programming language, for the most part. Using sed for anything substantial requires you to think in strange directions, to find ways to thread state and control flow through the tiny holes the language gives you.
- WastingMyTime89 5y ago
  I think I both agree and disagree. As you rightfully pointed, I wouldn’t do anything overly complicated with sed because it really is a line editor and anything which is not line based and can’t be explained simply in the context of using an editor will quickly seem clunky and overly complicated. I wouldn’t do anything period with awk. I agree with you that it’s just another programming language but one which feels extremely dated with both a terrible syntax and awkward semantics. The rare times I have encountered something done with awk it was either so simple it should have been done with sed or so complicated I wish a proper scripting language like python or perl would have been used instead. As far as I’m concerned, awk is a tool without a proper use-case. It’s always better to use something else.

visualphoenix 5y ago

This is awkward… I know it’s hip and cool to ask for donations, but for a tiny collection of sed scripts which aren’t including detailed explanations? I feel like sed is what deserves the donation… http://sed.sourceforge.net/sed1line.txt http://sed.sourceforge.net/sed1line.txt ? https://www.fsf.org/about/ways-to-donate/ https://www.fsf.org/about/ways-to-donate/

adrianscheff 5y ago
Hey there! Thank you for the input! I'm sorry you feel that asking for donations is unwarranted. I agree wholeheartedly that FSF deserves all the love it can get. I'm not asking to get rich or take the spotlight - I'm asking so I can keep improving that guide & create more. I didn't added explanations (although I wanted too) since this was intended as a quick tips page for sed. I think there are much better guides than I could ever make (including the info page). In a way it works better since it's more digestible. Thank you for the link you provided - it looks awesome! I'll look into it and append the existing guide if needed (while giving credits of course)

krinchan 5y ago

I use sed -ibak ‘10d’ ~/.ssh/.known_hosts so many times a week. awk, sed, and regex really are the two programs I am happy to have learned well at my first job. They’ve helped me immensely throughout my 10 years as someone who bridges developer and ops roles.

mdaniel 5y ago
For that one specifically, you may want to check your copy of "ssh-keygen" as it has recently(?) learned to manage those known_hosts entries without resorting to text file manipulation (`ssh-keygen -r something.example.com` for example)

cjvirtucio 5y ago

I'm annoyed at how sed has --regexp-extended, while grep has `--extended-regexp`. I'm sure there's a reason for it, but it always makes me stop and think which command has which flag.

jaimehrubiks 5y ago

What is the easiest way to make sure a "field, or option" is present in a file. Similar to what ansible's lineinfile does? For example suppose I need to add this setting to /etc/ssh/sshd_config: PermitRootLogin yes One idea that comes to mind is to first delete the line with the line shown in this post, and then add it back: Regex sed -E '/^#/d' file.txt - delete lines where regex matches sed -E '/^PermiteRootLogin/d' /etc/ssh/sshd_config ; echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config Is there a more straightforward way with sed?

pnutjam 5y ago
https://stackoverflow.com/questions/13014055/sed-command-how-to-replace-if-exists-else-just-insert https://stackoverflow.com/questions/13014055/sed-command-how...

jagged-chisel 5y ago

> Print one line > > sed -n '10p' myfile.txt Which line? Why p? I can make assumptions that this means “line 10, print” but easy-to-explain could go a small step further and complete the explanation.

max_hammer 5y ago

https://edoras.sdsu.edu/doc/sed-oneliners.html https://edoras.sdsu.edu/doc/sed-oneliners.html