4 ms·

Sed is one of those tools that once you learn it, you'll start to wonder how you ever got by without it. This is a great set of tutorials, he also wrote one ab

Sed is one of those tools that once you learn it, you'll start to wonder how you ever got by without it. This is a great set of tutorials, he also wrote one about Awk: http://www.grymoire.com/Unix/Awk.html http://www.grymoire.com/Unix/Awk.html Get to know these two tools and you'll be amazed at the hours you can save and what you can do, especially with text files.

Fede_V 12y ago
I am always amazed by how much people can get done using piped unix commands - but, personally, I find it much easier to just use sed when I need to quickly edit a few files, and if I need to do anything slightly more elaborate, to do it in Python with a script. It's obviously much slower - but I've never been in a position where I needed insane speed to quickly fix a bunch of files.
- uxcn 12y ago
  I'll sometimes use sed/awk/cut/etc... to get one off summary info out of large files. Excel/LibreOffice would choke, and loading it into a database for one off is painful.
  - csandreasen 12y ago
    I fully agree. I find it's worth keeping large files like these in a tab delimited format instead of CSV specifically so that I can run them through the standard Unix tools instead of mucking around with quoting issues and stray commas.
- sbilstein 12y ago
  The nice thing about these tools is you can count on them being on any unix machine w/o worry about installing them on a remote locked down box of some kind. You can also avoid any dependency complexity. They can be real awkward to remember however and the man pages can sometimes be a handful, so I try to store common idioms in gists or in my .bash_profile whenever I develop one.
  - Anthony-G 12y ago
    With regard to cross-platform compatibility, it's worth knowing some of the basic differences between GNU sed and BSD sed. I like the GNU extensions -- particularly the extensions to regular expression such as non-printing characters (e.g. `\t`) and character classes such as `\w` and `\b`. I almost always use the GNU `-i, --in-place` option once I'm satisfied that my sed commands do what I want. A couple of years ago, I was using my other half's Mac (OS X 10.4 with a very old version of BSD sed) and I really missed the GNU extensions. When you say you store common idioms in your Bash profile, do you mean storing the commands as Bash aliases or functions? I have similar issues with remembering syntax and building sed commands and I've been trying to do something similar to avoid spending time building a complex command from scratch when I already created a similar one some time previously.
    - sbilstein 12y ago
      I usually store things as aliases...and while sometimes I don't end up using the alias exactly having a useful name that describes what it does helps me tailor the command later on. For example I have the following alias 'watch_port' that looks like this: sudo ngrep -W byline -d en6 -qilw 'get|post' tcp dst port so `watchport 8080` will print all network traffic over port 8080 on my ethernet. I actually rarely use this as 'watch_port' directly, but it helps me remember quickly how to bend ngrep to my needs.
      - Anthony-G 12y ago
        Thanks for the response. I used to keep a list of long commands that I had constructed in a plain text file, titled `useful-commands.txt` but that became too unwieldy. Now, similar to you, I try to store them as aliases – even if I don’t use the command exactly as it was saved. The hard part is coming up with a good, succinct name (descriptive but not too long) for the alias. For a long time, I didn’t like using aliases because I didn’t want to become overly reliant on my custom aliases – and then miss them when working on an unfamiliar system. This generally worked out alright when I was able to use `Ctrl-R` with a large Bash history. Now, I think that was an irrational rationale and that aliases are very useful shell features. I’m currently trying to organise my aliases and functions into useful groups such as `home_aliases.sh`, `cygwin_aliases.sh`, etc. so that they can be loaded as needed. I then plan on adding them to a git repository so that they can easily be used – and updated – on different systems. BTW, thanks for letting me know about ngrep. It looks like a useful complement to tcpdump.
sqrt17 12y ago
Maybe it's because Perl was really popular at the time I discovered Unix and its tool, but why would you use sed and awk instead of a Perl one-liner? (Or even :s// in vim or M-x query-replace-regexp in emacs, if it's just regex munging)
- emmelaich 12y ago
  Exactly. Use Perl (or maybe Python). sed is bullshit. Do you know it is not actually possible to get output from sed that does not contain a newline?
  - Tobu 12y ago
    Are you sure? sed has -z, it's useful.
- boomlinde 12y ago
  Perl isn't included in e.g. BusyBox. The additional overhead of including perl in an embedded distribution could be a valid reason for using sed where you could otherwise have used perl.
- Tobu 12y ago
  vim's regex syntax is unusual and sucky. Thankfully running commands is super-easy: :%!sed …

Fede_V 12y ago

I am always amazed by how much people can get done using piped unix commands - but, personally, I find it much easier to just use sed when I need to quickly edit a few files, and if I need to do anything slightly more elaborate, to do it in Python with a script. It's obviously much slower - but I've never been in a position where I needed insane speed to quickly fix a bunch of files.

uxcn 12y ago
I'll sometimes use sed/awk/cut/etc... to get one off summary info out of large files. Excel/LibreOffice would choke, and loading it into a database for one off is painful.
- csandreasen 12y ago
  I fully agree. I find it's worth keeping large files like these in a tab delimited format instead of CSV specifically so that I can run them through the standard Unix tools instead of mucking around with quoting issues and stray commas.
sbilstein 12y ago
The nice thing about these tools is you can count on them being on any unix machine w/o worry about installing them on a remote locked down box of some kind. You can also avoid any dependency complexity. They can be real awkward to remember however and the man pages can sometimes be a handful, so I try to store common idioms in gists or in my .bash_profile whenever I develop one.
- Anthony-G 12y ago
  With regard to cross-platform compatibility, it's worth knowing some of the basic differences between GNU sed and BSD sed. I like the GNU extensions -- particularly the extensions to regular expression such as non-printing characters (e.g. `\t`) and character classes such as `\w` and `\b`. I almost always use the GNU `-i, --in-place` option once I'm satisfied that my sed commands do what I want. A couple of years ago, I was using my other half's Mac (OS X 10.4 with a very old version of BSD sed) and I really missed the GNU extensions. When you say you store common idioms in your Bash profile, do you mean storing the commands as Bash aliases or functions? I have similar issues with remembering syntax and building sed commands and I've been trying to do something similar to avoid spending time building a complex command from scratch when I already created a similar one some time previously.
  - sbilstein 12y ago
    I usually store things as aliases...and while sometimes I don't end up using the alias exactly having a useful name that describes what it does helps me tailor the command later on. For example I have the following alias 'watch_port' that looks like this: sudo ngrep -W byline -d en6 -qilw 'get|post' tcp dst port so `watchport 8080` will print all network traffic over port 8080 on my ethernet. I actually rarely use this as 'watch_port' directly, but it helps me remember quickly how to bend ngrep to my needs.
    - Anthony-G 12y ago
      Thanks for the response. I used to keep a list of long commands that I had constructed in a plain text file, titled `useful-commands.txt` but that became too unwieldy. Now, similar to you, I try to store them as aliases – even if I don’t use the command exactly as it was saved. The hard part is coming up with a good, succinct name (descriptive but not too long) for the alias. For a long time, I didn’t like using aliases because I didn’t want to become overly reliant on my custom aliases – and then miss them when working on an unfamiliar system. This generally worked out alright when I was able to use `Ctrl-R` with a large Bash history. Now, I think that was an irrational rationale and that aliases are very useful shell features. I’m currently trying to organise my aliases and functions into useful groups such as `home_aliases.sh`, `cygwin_aliases.sh`, etc. so that they can be loaded as needed. I then plan on adding them to a git repository so that they can easily be used – and updated – on different systems. BTW, thanks for letting me know about ngrep. It looks like a useful complement to tcpdump.

sqrt17 12y ago

Maybe it's because Perl was really popular at the time I discovered Unix and its tool, but why would you use sed and awk instead of a Perl one-liner? (Or even :s// in vim or M-x query-replace-regexp in emacs, if it's just regex munging)

emmelaich 12y ago
Exactly. Use Perl (or maybe Python). sed is bullshit. Do you know it is not actually possible to get output from sed that does not contain a newline?
- Tobu 12y ago
  Are you sure? sed has -z, it's useful.
boomlinde 12y ago
Perl isn't included in e.g. BusyBox. The additional overhead of including perl in an embedded distribution could be a valid reason for using sed where you could otherwise have used perl.
Tobu 12y ago
vim's regex syntax is unusual and sucky. Thankfully running commands is super-easy: :%!sed …