8 ms·

Practical Shell Patterns I Use

• seanhunter 5y ago
  As an aside, "zwischenzug" is a great name. A "zwischenzug" is an "inbetween move" in chess that might happen in the middle of a sequence, and can often lead to unexpected results if one side hasn't been precise in their move order. https://www.chessjournal.com/zwischenzug/ https://www.chessjournal.com/zwischenzug/ for example has some more info if you're curious.
• pizza234 5y ago
  > ps -ef | grep VBoxHeadless | awk '{print $2}' | xargs kill -9 There's pkill for that :) pkill -KILL VBoxHeadless (the above assumes that the binary is called `VBoxHeadless`)
  • cryptonector 5y ago
    Generally you might want to use the `-x` argument to `pkill`.
  • kragen 5y ago
    Or killall.
  • deckard1 5y ago
    let me introduce you to TASK_COMM_LEN $ pgrep systemd-journald $ and: $ pgrep systemd-journal 431 $ Guess how long it took me to figure this out. This affects killall, pkill, and pgrep. You can use the "-f" flag with pkill/pgrep.
  • inetknght 5y ago
    kill -9 $(ps aux | grep .exe | sed -r 's/ +/ /g' | cut -d ' ' -f 2) I fucking hate wine. I wouldn't use it at all if it weren't for steam. And this is the only way to kill a wine process.
    • johnny22 5y ago
      it's easier to use pkill -9 -f <pattern> usually. depending on the goal here.
• kstenerud 5y ago
  After almost 30 years and tens of thousands of LOC, possibly even into 6 figures, I've now thrown in the towel and just use Python for all my scripting needs. I've yet to have a single shell script that hasn't turned into a maintenance nightmare, regardless of "bash best practices" (such as they are). Bash is a terrible language, and needs to die.
  • SkyMarshal 5y ago
    I've occasionally wondered why Python isn't the default scripting language on *nix by now. It's almost always installed by default, is more capable than bash, is dynamic/interpreted, has a repl, and seemingly everything else you would want in a Bash replacement. Is it just Bash's momentum, or something else that keeps bash hanging around?
    • mdaniel 5y ago
      Because shell is really good at gluing together commands and has reasonably straightforward error management mechanisms Contrast that with: subprocess.run("some_thing with-output") subprocess.run("the_next_thing") which requires that the caller take steps to grab, check, and then re-emit any errors (or even just the "normal" output) produced by either of those, not to mention the horrors of output buffering for long-running processes and the ever-present encoding woes since subprocess interfaces as bytes not str I guess it is the same situation with all programming -- diligent programmers can make it work, less disciplined users inflict pain upon all downstream users of poorly coded scripts
    • enriquto 5y ago
      Common shell tasks (piping, process substitution, create a string with the output of a program, creating and redirecting output and input to files) are extremely cumbersome in python.
    • nixpulvis 5y ago
      Please show me how to do `ls | grep -i old` in Python? For a short while I was following https://github.com/matz/streem https://github.com/matz/streem (from the original author of Ruby) with interest in possibly being adapted to superseed Bash. I have no idea how realistic that actually is. I also use fish a lot, but it has a lot of it's own problems. The thing about all these "shells" that really seperates them from Python (et all) is that they lookup commands from PATH and execute them directly. Anything else is simply not a shell language no matter how great it is at scripting.
• gmuslera 5y ago
  I try to use "smaller" commands instead of more versatile but complex alternatives. I'm not sure if I get better cpu/io/etc usage by using cut or tr instead of sed or awk (or perl and so on) but it is a pattern I followed by decades, at least for simple enough commands. And I may have something less to remember if I do i.e. rev | cut | rev instead of awk to get the last field. And having something less to remember, and lowering the complexity for the reader (that not always be me, or if I am, may not be in the same mind state as when I wrote that) is usually good. But it also is having predictable/consistent patterns.
• gbrown_ 5y ago
  I know there are many ways the same thing can be done in the shell but there are so many problems here. Please take this as feedback and not harsh criticism (I know there's comment section on the blog but I'd rather not give my e-mail to yet another website). > cat /etc/passwd | sed 's/\([^:]*\):.*:\(.*\)/user: \1 shell: \2/' Besides the needless cat this is clearer in awk, a tool you mention just prior. awk '{FS=":"} {print "user: " $1, "shell: " $NF}' /etc/passwd > $ ls | xargs -t ls -l Beware files with whitespace in their names. > find . | grep azurerm | grep tf$ | xargs -n1 dirname | sed 's/^.\///' > > ... > > for each of those files, removes the leading dot and forward slash, leaving the final bare filename (eg somefile.tf) No it doesn't it returns the names of parent directories where files ending in "tf" are found and where the path also includes the string "azurerm". > $ ls | grep Aug | xargs -IXXX mv XXX aug_folder Ew. mv *Aug* aug_folder/ Though now the issue of whitespace in path names is mentioned. Another minior point here is that with GNU xargs at least when using -I -L 1 is implied, so the original example is equivalent to a loop. > $ ps -ef | grep VBoxHeadless | awk '{print $2}' | xargs kill -9 pkill VBoxHeadless You acknowledge to avoid SIGKILL if possible so I don't know why you put it in the example. > $ sudo updatedb > $ sudo locate cfg | grep \.cfg$ locate '*.cfg' No idea why sudo is used when running locate, also it takes a glob so the grep here can be avoided. > $ grep ^.https doc.md | sed 's/^.(h[^])]).*/open \1/' | sh Now this is just nutty. > This example looks for https links at the start of lines in the doc.md file No it doesn't, it matches the start of the line, followed by a character, then "https". The sed then matches start of the line, followed by a character, starts a capture group of the letter "h" followed by a single character that is not "]" or ")", closes the capture group and continues to match anything. The example given will always result in "open ht" for any hit. Then there's the piping to sh. You mention like to drop this to review the list. I'd suggest focusing on extracting the list of links then pipe to xargs open. If you get a pipeline wrong you could unintentionally blast something into a shell executing anything which might be very bad. > Give Me All The Output With 2>&1 &> stdout-and-stderr But each to their own. > env | grep -w PATH | tr ':' '\n' echo $PATH | tr ':' '\n' or tr ':' '\n' <<< $PATH Edit: I know the formatting of this comment is terrible, I think quoting the sed example caused the rest of the comment to be marked up as italics so I indented it along with the other quotes as code.
  • 5y ago
• bxparks 5y ago
  Interesting stuff. One small correction that I noticed: $ sudo locate cfg | grep \.cfg$ should contain quotes, like this: $ sudo locate cfg | grep '\.cfg$' because the shell swallows the backslash. But locate(1) supports globs, so I think this could be simplified to just: $ sudo locate '*.cfg' But on my Ubuntu, /etc/updatedb.conf has PRUNE_BIND_MOUNTS="yes", so my /home, on a separate partition, does not get updated. So I often resort to: $ find -name '*.cfg' somewhere in my $HOME directory.
  • nuxi 5y ago
    You can create a separate locate database for /home if you want. Set it up for periodic updates (via cron or other means) and then use "locate -d <path_to_db>" for searching.
  • revscat 5y ago
    Or using globbing in zsh: $ echo **/*.cfg You can even order my most recently modified: $ echo **/*.cfg(om) `setopt EXTENDED_GLOB` needs to be on.
  • twic 5y ago
    You might want to add -b: -b, --basename Match only the base name against the specified patterns. This is the opposite of --wholename. To ignore files which have .cfg somewhere in their path, but aren't called .cfg. Or you might not!
• oandrew 5y ago
  bash 4 supports `|&` as an alternative to `2>&1` which looks better in pipelines e.g. $ docker logs container |& grep word
  • LanternLight83 5y ago
    I was looking for this just the other day, but thought it would have been the other way around (&|, consistent with &>), and moved on with 2>&1 |
• oandrew 5y ago
  One feature I really wish for in shells is something similar to Perl's unquoted strings / single line concise HEREDOC syntax / raw literal syntax. e.g. $ echo q(no ' escaping ` required " here) no ' escaping ` required " here This would make typing sql / json much easier. To my knowledge none of the shells implement this. Does anyone know why?
  • chasil 5y ago
    The POSIX shell was set in stone in the early '90s. The standards board actually removed features from the Korn shell in order for Xenix-286 or comparable systems to be able to run it in a 64k text segment, with clean and maintainable C (ksh88 is very ugly C). The standards for the POSIX shell are controlled by the Austin group/OSF, and they are not receptive to changes, unfortunately.
• honkycat 5y ago
  Going to echo: Bash is bad. Bash is a feature-impoverished language and belongs in the dustbin. I don't understand why we script with one hand tied behind our back. With Python I can install "click" and get great argument parsing and multi-command support with a simple install. Bash's argument parsing is much more verbose and arcane. Sure, it has nice ways to combine tools and parse strings and such. However: You could also implement those nice abstractions in a higher quality language. So a good thing about bash does not cancel out all the bad. I would LOVE a toolkit that is: - A single binary - That parses a single file like make ( but NOT a makefile ) - That uses a well known, powerful language. - That lets me declare tasks, validations ( required arguments, etc ), and workflows as a first-class citizen - With out of the box `--help` support - That lets me import other makefile-like files to improve code re-use ( both remote and local )
  • t43562 5y ago
    I think the features that it has make it the language that it is. If you try to remove the need for combining other programs by making everything builtin you wouldn't design it the way it is.
  • eternityforest 5y ago
    It feels like trying to program in an esoteric language, or a toy language made by a 16yo. It's nice if you have no logic or loops or variables, but just goes downhill from there. I really don't know why we can't just take a fork of MicroPython, agree that we are not gonna make any backwards incompatible changes anytime this decade, and none at all without changing the name, and then just include it everywhere. All these "Let's make a better shell" projects are still held back by being shell. They are trying to be a UI and a language at once, and they're just OK at UI and kinda crappy at programming. I think the closest tool to what you describe is Ansible which is in fact wonderful.
• t43562 5y ago
  The one that I find repeatedly useful: find . -name <some pattern> | { while read F; do <do something to file F>; done; } This lets you do things of almost any level of complexity to files or data listed as input and it doesn't require temporary files so you can process a lot of things. e.g. you might use sed -i to alter the files in some way or you might find media files of some kind into ffmpeg to convert them.
  • chasil 5y ago
    If you really had a lot of ffmpeg work to do, then GNU xargs can let you run the jobs in parallel, one for each of your available CPUs. You can even adjust the number of processes up or down by sending the xargs parent the appropriate signals. https://www.linuxjournal.com/content/parallel-shells-xargs-utilize-all-your-cpu-cores-unix-and-windows https://www.linuxjournal.com/content/parallel-shells-xargs-u...
    • t43562 5y ago
      xargs is a fantastic tool - there's another similar one, think called GNU parallel which is worth a look: https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/
  • geocrasher 5y ago
    You can use another construct built into find that negates the need for a for loop: find . -name foo -exec do something to {} \; For example: find . type f -exec chmod 644 {} \; find . -size +1G -exec ls -lh {} \; |awk '{ print $5" "$9 }'
    • t43562 5y ago
      Yes, I think most of the time this is all you need. I just like the for loop because I can do filename manipulations in the loop and the input to the loop doesn't always have to be find - it can be "grep <something> -l ./" for example - so I use it even if it's verbose because it's only one thing to remember. Perhaps that is a problem with bash and the tools - there are so many features that take years to discover. Every few weeks I come across a new one.
• yakshaving_jgt 5y ago
  Being able to write a file directly in the shell with heredocs is so cool, but I know I'll never use it because it's far quicker to just open vim and type with all the commands and verbs and motions that I'm used to.
  • dtgriscom 5y ago
    Ah, but that only writes the file once. Heredocs can write a new file, in a different position and with different variable replacements, as many times as you like. Very powerful.
    • yakshaving_jgt 5y ago
      I’m not sure I follow. It’s pretty straightforward to write the contents of a vim buffer to a file.
• AceJohnny2 5y ago
  Can anyone recommend a modern explanation of sed's more advanced features? I'm ashamed that, despite having learned many other ostensibly more complex systems and languages, sed continues to elude me. Of course I get the usual 's/foo/bar/g' stuff with regular expressions and placeholders, but I still don't really understand most of the other commands or, more usefully, the addresses. Perhaps I need to spend quality time with the manpage in a cabin in the woods, away from modern distractions, but so far it has proven impermeable. Perhaps I'm missing a proper understanding of its 'pattern space' vs 'hold space'.
  • lhoursquentin 5y ago
    https://www.grymoire.com/Unix/Sed.html https://www.grymoire.com/Unix/Sed.html is a good resource to get started, though it's definitely something you should not just read due to the fact that sed code is pretty hard to read. Have a terminal on the side and try it out live. Once you understand pattern and hold spaces sed becomes a pretty fun esoteric language :)
  • JNRowe 5y ago
    Unlike the GNU sed manpage, the info docs are really quite thorough and contain many examples. They cover the vast majority of the features, and they call out GNU specific extensions so they're useful for learning non-GNU sed too. They're available in HTML too¹, if you don't like the info user experience. ¹ https://www.gnu.org/software/sed/manual/html_node/index.html https://www.gnu.org/software/sed/manual/html_node/index.html
    • AceJohnny2 5y ago
      Awesome! They are indeed much clearer than the FreeBSD manpage I have on macOS.
• bostonsre 5y ago
  > Quick Infinite Loop I use this a lot for polling type commands: do_every_n_secs "ps -eaf | grep blablabla" 5 function do_every_n_secs (){ cmd=$1 secs=$2 if [[ $secs -eq "" ]]; then secs=10 fi while true; do echo "################################################################"; date; eval $cmd sleep $secs; done } ```
  • deleted 5y ago
    [deleted]
  • JNRowe 5y ago
    In case you're not aware watch(1) performs basically that task: watch -n 5 "ps -eaf | grep blablabla". It can also highlight changes in the text, break itself out of the loop under conditions, and a few other things. The major difference between your function and watch being that watch will use the alternate screen for output if possible.
    • bostonsre 5y ago
      Nope, I was not aware. Neat.
  • tomcatfish 5y ago
    Is there a way to make `watch` do this instead? It always goofs up my output, but it would be cool if you could use it here.
  • chasil 5y ago
    The grep above will match itself in the output. One way that I have seen to filter it out is grep -v grep, but if you are actually looking for grep processes, then this won't work. The shell can also filter these out. Consider this basic shell function: pps () ( IFS=\| ps -ef | while read a do case $a in *$1*|+([!0-9])) printf %s\\n "$a";; esac done ) This uses a shell pattern instead of a regular expression (shells with more features than POSIX can also use regex). It's actually a dual pattern, either no numbers which match the header, or a match of the pattern specified as the first function argument. The function uses a subshell () instead of command grouping {}. This allows IFS to be manipulated without modifying this setting outside the function, at the cost of a fork. POSIX doesn't support local shell variables; if your shell does (with typeset or local keywords), it will be slightly more efficient. My function would slot into yours, but yours would not work in the Debian default (dash) shell, because the "function" keyword is not defined in POSIX. POSIX approaches are the most important, because you can use them just about everywhere.
• bsder 5y ago
  As much as I like these, these are also how you blow both feet off. As long as you're just exploring (everything is read only), you're okay. The moment "kill" and "rm" (anything mutable) come into the picture on a long shell pipeline, stop. You probably don't want to do that. You need to be thinking very hard to make sure that an errant glob or shell substitution isn't going to wipe out your system. Make a complicated pipeline that creates a simple file, examine the simple file by hand, and then feed that file to "kill"/"rm"/"whatever" as close to unmodified as possible. You may still blow your system away. However, at that point, you've given yourself the best chances to not do so.
• AzzieElbab 5y ago
  I have written thousands locs in bash but every time I have to write another one my teeth start to ache. Not sure why
• justsomehnguy 5y ago
  > Here’s a real-world example I used recently gci -rec | ? name -match zabb | ? Extension -eq .deb | % Name Of course it could be replaced with gci -rec | ? name -match zabb.+\.deb$ | % name And you can filter on any property (a little excerpt): Mode : --r-- BaseName : zabbix-release_5.0-1+bionic_all Length : 4240 DirectoryName : /home/appliance FullName : /home/appliance/zabbix-release_5.0-1+bionic_all.deb Extension : .deb Name : zabbix-release_5.0-1+bionic_all.deb CreationTime : 5/11/2020 12:07:35 PM > ps -ef | grep VBoxHeadless | awk '{print $2}' | xargs kill -9 Get-Process apach* | Stop-Process -Force But 'pOwErShEll iS sO vErBoSe' > Find Files Ending With… gci / -re | ? name -match \.deb$ Would take longer, sure, but still the same Get-ChildItem cmdlet, no need to remember another two utilities > $ env | grep -w PATH | tr ':' '\n' PS /home> ( env ) -match '^PATH' -replace ':',"`n" PATH=/opt/microsoft/powershell/7 /home/appliance/bin /home/appliance/.local/bin [stripped] Again, just built-in functions, no external utilities As usual - there is a ton of useful and nice QOL utils in *nix world, but it always feels like scratching left ear with the right hand.
  • majkinetor 5y ago
    It event gests way easier then that in PowerShell: > Get-Process apach* | Stop-Process -Force ps apach* | kill > gci -rec | ? name -match zabb.+\.deb$ | % name gci -rec | ? name -like zabb*.deb | % name
    • justsomehnguy 5y ago
      > ps apach* | kill Not in Linux, because it calls native binaries there. > gci -rec | ? name -like zabb*.deb | % name Yep, even simpler. Just the original example was with regex, so I made a comparable regex example. > $ env | grep -w PATH | tr ':' '\n' Today it occurred to me what you can just use split: ( env ) -match '^PATH' -split ':'
    • justsomehnguy 5y ago
      Linux is clearly rotting my mind. $env:PATH -split ':'
• bravetraveler 5y ago
  No mention of my pet peeve! You don't need a subshell/seq to count. The shell is smart, it can count for you - and even respect digits. These examples are subtle, but different in result: for i in {00..10} ; do echo $i ; done for i in {0..10} ; do echo $i ; done