22 ms·
Bash patterns I use weekly
- jph 5y ago> git bisect is the "real" way to do this, but it's not something I've ever needed git bisect is great and worth trying; it does what you're doing in your bash loop, plus faster and with more capabilities such as logging, visualizing, skipping, etc. The syntax is: $ git bisect run <command> [arguments] https://git-scm.com/docs/git-bisect https://git-scm.com/docs/git-bisect
- OskarS 5y agoYes, git bisect is the way to go: in addition to the stuff you mentioned, his method only dives into one parent branch of merge commits. git bisect handles that correctly. A gem of a tool, git bisect.
- ljm 5y agoBisect also does a binary search so if you're looking for one bad commit amongst many others, you'll find it much more quickly than linearly testing commits, one at a time, until you find a working one.
- randomopining 5y agoHow does bisect help in a large project? Seems like it would be best to use personal expertise to find it
- matt_kantor 5y agoI've only used bisect a handful of times, but I think it can be most useful in large projects with many contributors. In smaller projects you can spot a bug and often surmise "I bet this is related to Fred's pull request yesterday that updated the Foo module", but in a larger repository where you don't have all the recent history in your head you might not even know where to start looking. In such cases being able to binary search through history is handy.
- deleted 5y ago[deleted]
- ljm 5y agoThis is a hilariously absurd proposition. The existence of `git bisect` necessarily precludes an issue that personal expertise cannot find without some help. Personal expertise tells you to use a tool like git bisect to find a problem, so you are more effective in your work. The same as it tells you to use gdb to debug a stack trace. Or do you eyeball the CPU executing instructions in realtime? An absence of personal expertise will convince you that you are smart enough to do it on your own.
- randomopining 5y agoSo let’s say there’s 50 commits in a short time between good and bad. Of those 50 there’s a few that are “maybes”. Bisect is completely useless
- MainJane 5y agogit-bisect is nice if you are looking for a git commit. If you are looking for a limit or the failing part of a file have a look at: https://gitlab.com/ole.tange/tangetools/-/tree/master/find-first-fail https://gitlab.com/ole.tange/tangetools/-/tree/master/find-f...
- geocrasher 5y agoI want to like this, but the for loop is unnecessarily messy, and not correct. for route in foo bar baz do curl localhost:8080/$route done That's just begging go wonky. Should be stuff="foo bar baz" for route in $stuff; do echo curl localhost:8080/$route done Some might say that it's not absolutely necessary to abstract the array into a variable and that's true, but it sure does make edits a lot easier. And, the original is missing a semicolon after the 'do'. I think it's one reason I dislike lists like this- a newb might look at these and stuff them into their toolkit without really knowing why they don't work. It slows down learning. Plus, faulty tooling can be unnecessarily destructive.
- masklinn 5y agoSeems like you could say the same thing about every snippet e.g. running jobs is the original purpose of a shell, (3) can be achieved using job specifications (%n), and using the `jobs` command for oversight.
- lambic 5y agoEven more correct would be to use an array: stuff=("foo foo" "bar" "baz") for route in "${stuff[@]}"; do curl localhost:8080/"$route" done
- gcmeplz 5y ago"${stuff[@]}" would be turning it back into "foo bar baz" though, right? I think if you were using arrays for this, it'd be something like: stuff=("foo" "bar" "baz"); array_length=${#stuff[@]}; for i in $(seq 0 $array_length); do curl localhost:8080/${stuff[i]} done I'm betting even that isn't right: as soon as bash arrays are a thing, I reach for a different language. [edit]: trying to get formatting correct
- computronus 5y ago"${stuff[@]}" expands into the elements of the array, with each one quoted - so even that first "foo foo" element with a space will be handled correctly. That is how I would write the loop as well, in general. However, your technique also works, with some tweaks: * The loop goes one index too far, and can be fixed with seq 0 $(( array_length - 1 )) * There should be quotes around ${stuff[i]} in case it has spaces
- R0flcopt3r 5y agoYou can use `wait` to wait for jobs to finish. some_command & some_other_command & wait
- masklinn 5y agoOne issue with that is it won't reflect the failure of those commands. In bash you can fix that by looping around, checking for status 127, and using `-n` (which waits for the first job of the set to complete), but not all shells have `-n`.
- guruparan18 5y agoI am confused how this works. I would assume `SECONDS` would just be a shell variable and it was first assigned `0` and then it should stay same, why did it keep counting the seconds? > SECONDS bash: SECONDS: command not found > SECONDS=0; sleep 5; echo $SECONDS; 5 > echo "Your command completed after $SECONDS seconds"; Your command completed after 41 seconds > echo "Your command completed after $SECONDS seconds"; Your command completed after 51 seconds > echo "Your command completed after $SECONDS seconds"; Your command completed after 53 seconds
- gcmeplz 5y ago`SECONDS` is like `PWD`: the shell keeps track of updating it somewhere, and it'll tell you how long your shell has been running. https://www.oreilly.com/library/view/shell-scripting-expert/9781118166321/c03-anchor-3.xhtml https://www.oreilly.com/library/view/shell-scripting-expert/...
- guruparan18 5y agoThank you. That make sense now.
- ghostly_s 5y agoHuh, never knew about $SECONDS.
- usefulcat 5y ago> Use for to iterate over simple lists I definitely use this all the time. Also, generating the list of things over which to iterate using the output of a command: for thing in $(cat file_with_one_thing_per_line) ; do ...
- phone8675309 5y agoWhy not? xargs -n1 command < file_with_one_thing_per_line
- make3 5y agothat's like bash 101, not sure why it's in there
- barbazoo 5y ago> git bisect is the "real" way to do this, but it's not something I've ever needed uh, yeah, you did need it, that's why you came up with "2. Track down a commit when a command started failing". Seriously though, git bisect is really useful to track down that bug in O(log n) rather than O(n).
- masklinn 5y agoAlso provides super useful commands like skipping commits because some of your colleagues are assholes and commit non-working code.
- deckard1 5y agoFor the given command, if the assumption is the command failed recently, it's likely faster than bisect. You can start it and go grab a coffee. It's automatic. I wish my usage of bisect were that trivial, though. Usually I need to find a bug in a giant web app. Which means finding a good commit, doing the npm install/start dance, etc. for each round.
- DylanSp 5y agoAs long as you can detect your bug from a script, you can pass the script to git bisect run and it'll run automatically.
- phist_mcgee 5y agoI've had to git bisect a react native app in a monorepo before. Took me nearly a day to track down the commit, due to the size of node_modules and the need to do a `pod install` every time.
- Izkata 5y ago> and the need to do a `pod install` every time. Phrasing makes it sound like you were doing it manually, so if you didn't know, you can just drop a script in the directory, say, "check.sh": #!/bin/bash if ! pod install; then exit 125 fi do-the-actual-test make it executable, and: git bisect run ./check.sh The special exit code 125 tells bisect "this revision can't be tested, skip it", otherwise it just uses exit code 0 for "good" and 1-127 (except for 125) for "bad".
- bloopernova 5y agoThis thread seems like a good place to ask this: When you're running a script, what is the expected behaviour if you just run it with no arguments? I think it shouldn't make any changes to your system, and it should print out a help message with common options. Is there anything else you expect a script to do? Do you prefer a script that has a set of default assumptions about how it's going to work? If you need to modify that, you pass in parameters. Do you expect that a script will lay out the changes it's about to make, then ask for confirmation? Or should it just get out of your way and do what it was written to do? I'm asking all these fairly basic questions because I'm trying to put together a list of things everyone expects from a script. Not exactly patterns per se, more conventions or standard behaviours.
- mason55 5y agoIMO any script that makes any real changes (either to the local system or remotely) should take some kind of input. It's one thing if your script reads some stuff and prints output. Defaulting to the current working directory (or whatever makes sense) is fine. If the script is reading config from a config file or envvars then it should still probably get some kind of confirmation if it's going to make any kind of change (of course with an option to auot-confirm via a flag like --yes). For really destructive changes it should default to dry run and require an explicit —-execute flag but for less destructive changes I think a path as input on the command line is enough confirmation. That being said, if it’s an unknown script I’d just read it. And if it’s a binary I’d pass —-help.
- bloopernova 5y agoThanks for the reply! I really appreciate being able to pick other folks' brains on here :)
- gmuslera 5y agoWhat is your intended audience? It is you? A batch job or called by another program? Or a person that may or not be able to read bash and will call it manually? A good and descriptive name comes first, then the action and the people that may have to run it are next.
- michaelhoffman 5y agoI have something like this in my bashrc: preexec () { # shellcheck disable=2034 _CMD_START="$(date +%s)" } trap 'preexec; trap - DEBUG' DEBUG PROMPT_COMMAND="_CMD_STOP=\$(date +%s) let _CMD_ELAPSED=_CMD_STOP-_CMD_START if [ \$_CMD_ELAPSED -gt 5 ]; then _TIME_STR=\" (\${_CMD_ELAPSED}s)\" else _TIME_STR='' fi; " PS1="\n\u@\h \w\$_TIME_STR\n\\$ " PROMPT_COMMAND+="trap 'preexec; trap - DEBUG' DEBUG" Whenever a command takes more than 5 s it tells me exactly how long at the next prompt. I didn't know about `$SECONDS` so I'm going to change it to use that.
- masklinn 5y agoFWIW that is supported by many zsh themes like pure, p9k, p10k, ... (and often enabled by default). Also: REPORTTIME If nonnegative, commands whose combined user and system execution times (measured in seconds) are greater than this value have timing statistics printed for them. which is slightly different but generally useful, and built-in.
- spondyl 5y agoCame here to +1 REPORTTIME for zsh
- oweiler 5y agoBash 5.0 also has $EPOCHREALTIME
- caymanjim 5y agoInstalling GNU stuff with the 'g' prefix (gsed instead of sed) means having to remember to include the 'g' when you're on a Mac and leave it off when you're on Linux, or use aliases, or some other confusing and inconvenient thing, and then if you're writing a script meant for multi-platform use, it still won't work. I find it's a much better idea to install the entire GNU suite without the 'g' prefix and use PATH to control which is used. I use MacPorts to do this (/opt/local/libexec/gnubin), and even Homebrew finally supports this, although it does it in a stupid way that requires adding a PATH element for each individual GNU utility (e.g. /usr/local/opt/gnu-sed/libexec/gnubin).
- l0b0 5y ago> 1. Find and replace a pattern in a codebase with capture groups > git grep -l pattern | xargs gsed -ri 's|pat(tern)|\1s are birds|g' Or, in IDEA, Ctrl-Shift-r, put "pat(tern)" in the first box and "$1s are birds" in the second box, Alt-a, boom. Infinitely easier to remember, and no chance of having to deal with any double escaping.
- marginalia_nu 5y agoUsing an IDE kind of handicaps you to only working with your IDE though. The shell works everywhere for every use case.
- withinboredom 5y agoThat particular IDE works on Windows though… no idea how to use Powershell…
- marginalia_nu 5y agoWSL2?
- withinboredom 5y agoFrom experience, WSL is pretty slow vs. native on fs operations.
- marginalia_nu 5y agoFor sure, but for the convenience of being able to use bash commands on Windows it's well worth it.
- l0b0 5y agoI've heard this many times, and I don't really understand the argument. I've used the shell plenty of times for this sort of work, but it's much more complicated to do certain things correctly than in an IDE. Things like looping over any filename, escaping arbitrary strings for use within sed patterns, xargs, multi-line replacements and regex lookahead require quite some in-depth knowledge, trial and error, and sometimes installing special tools because the defaults don't support the relevant patterns (like path NUL terminators or lookaheads). I don't have to deal with any of these in IDEA.
- stewartbutler 5y agoI've always had trouble getting `for` loops to work predictably, so my common loop pattern is this: grep -l -r pattern /path/to/files | while read x; do echo $x; done or the like. This uses bash read to split the input line into words, then each word can be accessed in the loop with variable `$x`. Pipe friendly and doesn't use a subshell so no unexpected scoping issues. It also doesn't require futzing around with arrays or the like. One place I do use bash for loops is when iterating over args, e.g. if you create a bash function: function my_func() { for arg; do echo $arg done } This'll take a list of arguments and echo each on a separate line. Useful if you need a function that does some operation against a list of files, for example. Also, bash expansions (https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html https://www.gnu.org/software/bash/manual/html_node/Shell-Par...) can save you a ton of time for various common operations on variables.
- jolmg 5y ago> I've always had trouble getting `for` loops to work predictably, so my common loop pattern is this: The problem with the pipe-while-read pattern is that you can't modify variables in the loop, since it runs in a subshell.
- edgyquant 5y agoIt’s a trade off. I had to use a piped loop earlier this year to extract fallout new Vegas mods and textures since they all had spaces in their file names. For this it was perfect to pipe a list of the names to a loop, but for 99% of things I just use a for loop
- jolmg 5y agoYup. Nearly everything has tradeoffs. BTW, the problem I mentioned earlier can be avoided by using `< <()`: $ x=1 $ seq 5 | while read n; do (( x++ )); done $ echo $x 1 $ while read n; do (( x++ )); done < <(seq 5) $ echo $x 6 Almost makes me wonder what the benefit of preferring a pipe here is. I guess it's just about not having to specify what part of the pipeline is in the same shell.
- marcodiego 5y agoWe need a simpler regex format. One that allows easy searching and replacing in source code. Of course, some IDE's already to that pretty well, but I'd like to be able to do it from the command line with a stand alone tool I can easily use in scripts. The simplest thing I know that is able to do that is coccinelle, but even coccinelle is not handy enough.
- f0e4c2f7 5y agoIt's not perfect but I like sed for this.
- marginalia_nu 5y agoI think what we really need is one regex format. You have POSIX, PCRE, and also various degrees of needing to double-escape the slashes to get past whatever language you're using the regex in. Always adds a large element of guesswork even when you are familiar with regular expressions.
- Something1234 5y agohttps://regex101.com/ https://regex101.com/
- marginalia_nu 5y agoYeah I know all these dialects, the question is, which is the one I'm supposed to use for (this tool)?
- yesenadam 5y ago> I think what we really need is one regex format. See https://xkcd.com/927/ https://xkcd.com/927/ How standards proliferate
- unixhero 5y agoA lot of genious moves here I have never seen or thought of. Brilliant. Grabbing pids was mindblowingly effective.
- Hikikomori 5y agoMy daily bash pattern. cat file cat file | grep something
- Eduard 5y agohttps://security.stackexchange.com/questions/56307/can-cat-ing-a-file-be-a-potential-security-risk https://security.stackexchange.com/questions/56307/can-cat-i...
- phist_mcgee 5y agoI prefer less tbh, less pollution of the term output.
- amp108 5y agoYou could just `grep "something" file.
- penguin_booze 5y agoIndeed - it's a known anti-pattern[0]. [0] https://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat https://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat
- gnubison 5y agoIt’s simple enough to use macos sed here instead of needing gsed: just use ‘-i ''’ instead of ‘-i’.
- Arch-TK 5y agoThis post and comments section means I no longer wonder why 99% of shell scripts I come across look inept. I'm sorry guys but seriously please actually learn bash (and ideally not from this blog post). There's so many things wrong in the post and the comments that it's difficult to enumerate. To start with, if you ever feel the need to write a O(n) for loop for finding which commit broke your build, you DID need git-bisect. Definitely DONT'T wait for PIDs like that and if you do want to write code like that, maybe actually use the arrays bash provides? If your complaint is that for a in b c d; do ...; done is unclear, then maybe also use lists there, because what's definitely LESS clear is putting things in a variable and relying on splitting. And most importantly, DO quote things (except when it's unnecessary).
- gcmeplz 5y ago> Definitely DONT'T wait for PIDs like that and if you do want to write code like that, maybe actually use the arrays bash provides? What pattern would you recommend for waiting for PIDs / parallelizing commands & preserving exit codes? Fair point about arrays being a better fit rather than a string there.
- IndexPointer 5y agoTry GNU parallel.
- Arch-TK 5y agoSo in this case there's a couple notable things: `wait` can take no parameters this means that if you just ran a bunch of things in the background in your script and want to wait for all of them to finish, you don't need to track the PIDs or a loop, you can just `wait`. `wait` is a bash builtin (in this case) and as such it has no parameter limit (although I am told that actually there are some weird limits but it's very unlikely you will be able to spawn enough processes at once from a bash script to hit the limits). Given an array `pids` you can just do: `wait "${pids[@]}"` The only problem with the two above approaches is that in the former case, wait loses the return status and in the second case wait loses all but the status of the last ID you pass it, so the third option is: pids=() do_thing_1 & pids+=("$!") do_thing_2 & pids+=("$!") for pid in "${pids[@]}"; do wait "$pid" || status=$? done exit "${status-0}" Now you only have the issue left that this will report the LAST failing status.
- 1vuio0pswjnm7 5y agoTo conserve host resources RFC 2616 recommends making multiple HTTP requests over a single TCP connection ("HTTP/1.1 pipelining"). The cURL project said it never properly suported HTTP/1.1 pipelining and in 2019 it said it was removed once and for all. https://daniel.haxx.se/blog/2019/04/06/curl-says-bye-bye-to-pipelining/ https://daniel.haxx.se/blog/2019/04/06/curl-says-bye-bye-to-... Anyway, curl is not needed. One can write a small program in their language of choice to generate HTTP/1.1, but even a simple shell script will work. Even more, we get easy control over SNI which curl binary does have have. There are different and more concise ways, but below is an example, using the IFS technique. This also shows the use of sed's "P" and "D" commands (credit: Eric Pement's sed one-liners). Assumes valid, non-malicious URLs, all with same host. Usage: 1.sh < URLs.txt #!/bin/sh (IFS=/;while read w x y z;do case $w in http:|https:);;*)exit;esac; case $x in "");;*)exit;esac; echo $y > .host printf '%s\r\n' "GET /$z HTTP/1.1"; printf '%s\r\n' "Host: $y"; # add more headers here if desired; printf 'Connection: keep-alive\r\n\r\n';done|sed 'N;$!P;$!D;$d'; printf 'Connection: close\r\n\r\n'; ) >.http read x < .host; # SNI; #openssl s_client -connect $x:443 -ign_eof -servername $x < .http; # no SNI; openssl s_client -connect $x:443 -ign_eof -noservername < .http; exec rm .host .http;
- themk 5y agoThe curl binary will reuse the TCP connection when fed multiple URLs. Infact it can even use HTTP2 and make the requests in parallel over a single TCP connection. Common pattern I use is to construct URLs with a script and use xargs to feed to curl.
- 1vuio0pswjnm7 5y agoFor HTTP/1.1 pipelining, not HTTP/2 which not all websites support, the curl binary must be slower because the program tries to do more than just make HTTP from URLs and send the text over TCP. It tries to be "smart" and that slows it down. But dont't take my word for it, test it. For example, compare the retrieval speed of the above to something like sed -n '/^http/s/^/url=/p' URLs.txt|curl -K- --http1.1
- notatoad 5y agoI really love this style of blog post. Short, practical, no backstory, and not trying to claim one correct way to do anything. Just an unopinionated share that was useful to the author. It seems like a throwback to a previous time, but honestly can't remember when that was. Maybe back to a time when I hoped this was what blogging could be.
- js2 5y agoMy git-gsr (global search replace) command: https://gist.github.com/jaysoffian/0eda35a6a41f500ba5c458f02a29865d https://gist.github.com/jaysoffian/0eda35a6a41f500ba5c458f02... Uses perl instead of gsed, defaults to fixed strings but supports perl regexes, properly handles filenames with whitespace.
- penguin_booze 5y agoIf I find myself running a set of commands in parallel, I'd keep a cheap Makefile around: individual commands I want to run in parallel will be written as phony targets: all: cmd1 cmd2 cmd1: sleep 5 cmd2: sleep 10 And then make -j[n]
- harvie 5y agocurl localhost:8080/{foo,bar,baz}