12 ms·
No script is too simple
- jkubicek 6y agoI use fishshell, and I've gotten in the habit of creating a function called `t` to run tests. This function captures whatever test command I'm currently using. Just test one file, run one test case, use a debugger, capture coverage, etc. I don't save the function, so it doesn't persist across Terminal sessions. If I need to change how I'm running tests, I update the function. It's a small, but noticeable improvement over the way I was working before, either up-arrowing until I found the last time I ran tests, or typing, `pytest...` and letting autocomplete figure out what I was doing previously. edit: So yes, I am also a big fan of helping to enforce consistency by scripting even the small things.
- ch4s3 6y agoWould you mind sharing a gist of that function?
- dfinninger 6y agoI’m assuming they mean something like: t() { bazel test //some/specific/thing/... # or pytest, maven, sbt, etc etc } Add in whatever option you want to the test command there. Then you just press “t” and it runs your tests.
- Felk 6y ago> either up-arrowing until I found the last time I ran tests just as a substitute for up-arrowing, have you tried ctrl-r/reverse-i-search? (if that's a thing in fishshell) edit: I see wodenokoto already mentioned this workflow in another comment
- simongr3dal 6y agoin fish you can type some of the command and arrow-up. It will show only commands containing that substring.
- frutiger 6y agoSo does libreadline (and therefore bash). The following in ~/.inputrc does the trick for vi mode users: k: history-search-backward j: history-search-forward There is an equivalent for emacs mode.
- jonfw 6y agoCheck out FZF for fuzzy finding through your command history in this manner- it is absolutely the superior way to search command history
- Macha 6y agoIf you type before up arrowing this is what the up arrow does in fish
- wheybags 6y agoI wrote a small python script, that I alias in my shell to "b" (for build). When I run it in a given directory, it prompts me for a command, if it doesn't have one already saved. Subsequent runs just run the saved command, but it can save a different command for each folder. I use it to clear and remake my build directory using cmake on my c++ projects, with the various compile options saved in as well. It's basically a persistent version of what you describe. Here's the script, if you're interested. It's not super complex. https://github.com/wheybags/stuff/blob/master/build-dir-command.py https://github.com/wheybags/stuff/blob/master/build-dir-comm...
- bauerd 6y agoHave a function similiar to this, reruns the last test command function retest set --local cmd (history | grep -v history | grep -Em1 "((bundle exec)?rspec|go test|jest)") echo "Rerunning `$cmd`" eval $cmd end
- can16358p 6y agoI think that's the equivalent of abstraction/encapsulation in programming languages. Sometimes even one liners can be encapsulated in something else as the implementation might change but the purpose/role doesn't.
- chubot 6y agoShell functions can also accomplish the same purpose as tiny little scripts. For example, a 2 line shell script can be wrapped in a function instead: mymove() { cp --verbose $1 $2 rm --verbose $1 } Shell has good abstraction capacities! It even has some that other languages don't have: Shell Has a Forth-like Quality http://www.oilshell.org/blog/2017/01/13.html http://www.oilshell.org/blog/2017/01/13.html Pipelines Support Vectorized, Point-Free, and Imperative Style http://www.oilshell.org/blog/2017/01/15.html http://www.oilshell.org/blog/2017/01/15.html http://www.oilshell.org/blog/tags.html?tag=shell-the-good-parts#shell-the-good-parts http://www.oilshell.org/blog/tags.html?tag=shell-the-good-pa...
- jcynix 6y agoYes, shell functions are cool. I use them to augment standard commands too, e.g. make head(1) or tail(1) output more lines than usual, depending on the terminal's number of lines: head () { if [[ $# -eq 0 ]] then /usr/bin/head -$[(LINES-1)/2] elif [[ -f "$1" ]] then case $# in (1) /usr/bin/head -$[(LINES-1)/2] $* ;; (2) /usr/bin/head -$[LINES*5/12] $* ;; (3) /usr/bin/head -$[(LINES-1)/3] $* ;; (*) /usr/bin/head $* ;; esac else /usr/bin/head $* fi }
- brendyn 6y agowith that, I think if the cp fails, the rm will still be deleted. maybe 'set -euo pipefail' will fix it. Do the $1 and $2 need to be quoted too incase there are special characters?
- chubot 6y agoYes all that is true -- it was just a quick demonstration. By the way: - Oil has set -euo pipefail on by default. (And also nullglob) - Oil doesn't split words by default, so you don't need the quotes: https://www.oilshell.org/release/0.8.0/doc/simple-word-eval.html https://www.oilshell.org/release/0.8.0/doc/simple-word-eval.... Those are all "opt in" -- OSH still runs POSIX shell scripts, if you want all the extra ceremony :)
- choeger 6y agoConsider the task of doing anything with your software a program in a very specific DSL. Then the scripts are your verbs. What you also need are your nouns. For instance, deploying to a specific target could be "./scripts/deploy.sh stage", backporting a patch could be "./scripts/patch_release.sh 1.1.0 dbcde45", and creating a database migration script could be "./scripts/db_changed.sh 'add new field for model'" IMO thinking about the verbs is the first important step, but one should also always specify the nouns explicitly.
- danpalmer 6y agoI'm a big fan of this but I often find it in tension with necessary flexibility. The more flexibility I add to my scripts, the less useful they are for this purpose. I usually come down on: things where the output is shared should probably be scripts, things where it's just for me probably shouldn't be (so that I can be more flexible, and there will be less bit-rot of those scripts). Any other thoughts on this?
- chubot 6y agoYeah this is a good question. I lean toward making shell scripts work for a specific project. All the paths should be the same for all developers, so I usually use paths relative to the git repo root of the project. A key point is that when you want to "abstract", then the right way is usually to write a command line tool invoked from shell scripts in multiple projects. That is the natural way to get flexibility and reuse. But otherwise it's a bunch of commands dumped in one place. Examples: http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#semi-automation-with-runsh-scripts http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#... Here's a random example -- some scripts to generate source code as part of the build process: https://github.com/oilshell/oil/blob/master/build/codegen.sh https://github.com/oilshell/oil/blob/master/build/codegen.sh The paths are all hard-coded, which is a good thing. In my mind, the goal is to save time and reduce mistakes. And having a consistent dev environment between everybody on a project is almost a prerequisite for that, and shell can actually enforce that consistency! (i.e. the shell scripts don't work if people have quirks on their machine. They can check the environment too.)
- jason_zig 6y agoDid anyone else hold down shift + command and find the achievements part of the site? Fun idea...
- nicbou 6y agohttps://nicolasbouliane.com/achievements https://nicolasbouliane.com/achievements - for the curious
- wodenokoto 6y agoI have so many common one-liners I use in my current project (that I access using fuzzy search via ctr-R) that I'm thinking about having a file a'la "my-commands" and have it appended to my history, somehow. That would truly be the opposite of this advice. Maybe I need to think about this a bit more :)
- camnora 6y agoGreenclips [1] works well for this if you're a rofi [2] user. You can set a staticHistoryPath that points to a file. When activating Greenclips, you can search for the desired command. I've been using this on my Linux box for the last year or so and haven't looked back. [1] https://github.com/erebe/greenclip https://github.com/erebe/greenclip [2] https://github.com/davatorium/rofi https://github.com/davatorium/rofi
- sbt567 6y agoI'm using navi (https://github.com/denisidoro/navi https://github.com/denisidoro/navi) for commands that long enough and used less frequently. And it works great.
- lytedev 6y agoMakefile!
- dylan604 6y ago> have it appended to my history, somehow isn't that what .aliases is for?
- wodenokoto 6y agoI honestly don’t know.
- marcosdumay 6y agoShell aliases are your friend. just give them some memorable names, and add them to your .bashrc. Or, if they are very context sensitive (that's not great), there is a way to source a file every time you enter a directory, I just don't remember it.
- thenonameguy 6y agoI saw that the cost/benefit ratio of adopting https://github.com/casey/just https://github.com/casey/just in non-trivial projects was worth it as an alternative to bash scripts in script folders.
- Galanwe 6y agoSeems like a great little tool. Though it looks a bit too young for my taste: it's not available in most distro's base repositories yet, so it's going to be a tiny bit painful to deploy on every developer laptop, CI, and etc. I tend to prefer readily available tools like make, with 90% of the same features, but 100% distro coverage and previous developer knowledge.
- thenonameguy 6y agoThankfully we have solved this issue by adopting nix for setting up developer machines/project setup on top of your OS's package manager of choice (OSX/Linux). Additionally, you automatically get shell completion too! https://github.com/NixOS/nixpkgs/blob/3bb54189b0c8132752fff320a3bf441118119a5e/pkgs/development/tools/just/default.nix https://github.com/NixOS/nixpkgs/blob/3bb54189b0c8132752fff3... As described in the README, avoiding the 'build' part from Makefiles cut unnecessary complexity like .PHONY targets, which improves clarity. In smaller teams/companies it makes sense IMO.
- Galanwe 6y agoIt's a great habit. I use that same concept extensively for all my projects, whatever the language/stack, though I prefer to use Makefiles since I feel like it creates a more cohesive result. Very easy to chain, depend and read in the CI too.
- dahfizz 6y ago+1 for make. Its an amazing and underrated tool.
- jscheel 6y agoIt took me forever to come around on make, but I absolutely love it for things like this now.
- narwally 6y agoAny recommendations on learning to use make more effectively outside of just reading the docs? I feel like this is a glaring gap in my knowledge as I really only know the basics.
- makapuf 6y agoI'd say gnu make (dont use any other make) can be learned from the documentation, it readable. Also, it's super simple to try things yourself (unlike, say, k8s by example) : just use it in your project and use the doc as needed. I am very fond of make, to the point I did a video transcoding solution based on it and ffmpeg.
- TeMPOraL 6y agoRelated trick: GNU tools tend to come with Info pages - it's to man pages what a book is to a listicle. These Info pages can get you up to speed quickly, while giving you a deeper understanding of the tool at the same time.
- jscheel 6y ago
- deeg 6y agoHe mentions this in the original article: scripts can be functional documentation. They are an easy way to learn the common commands in a project, can call out the expected workflow, and document all the commands to accomplish it.
- chubot 6y agoYes, I use shell as executable documentation, and I think it's the best language for it. I outlined these ideas here: http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#semi-automation-with-runsh-scripts http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#... http://www.oilshell.org/blog/2020/07/blog-roadmap.html#more-posts-in-shell-the-good-parts http://www.oilshell.org/blog/2020/07/blog-roadmap.html#more-... As shown there, a lot of people are doing this under different names... I hope Oil can provide something consistent.
- cgarvis 6y agoI've been doing this with Makefiles for a while. It has always bugged me that I'm not "making" an artifact. Think I"m going to try this `run.sh` approach.
- chubot 6y agoYeah, as mentioned in the posts, make targets that are verbs rather than nouns should really be .PHONY, but most people forget that. While the idea of having Make's dependency engine is nice in theory, it falls down for one-off automation in my experience. For a couple reasons: (1) Make's dependency model has some well known deficiencies. It doesn't play well with tools that produce two files. It doesn't play well with tools that produce a directory tree of "dynamic" filenames (not known when you write the Makefile) (2) Most makefiles have bugs, especially when you do make -j (parallel builds). It's basically like writing a C program with a bunch of threads racing on global variables -- your Make targets will often be racing on the same file, leading to non-deterministic bugs. ----- So IMO it's better to mostly stick with the sequential model of shell for this kind of project automation. But shell can invoke make! When you know your dependencies, invoke them from run.sh! And when you're REALLY sure it's correct, invoke make -j :) In other words shell is my default, and make is only for when I want to spend the effort to write dependencies -- which is quite difficult, because Make provides you virtually no help with that. Bugs in dependencies are common and hard to find. If you're trained to run "make clean", then that's a symptom of a bug in the build specification. Shell "gets shit done" without these types of bugs. Debugging a shell script is very easy compared with debugging a makefile. The remaining problems with shell will hopefully be fixed by https://oilshell.org/ https://oilshell.org/ :) I do want to add some dependency support, but I didn't get to it: Shell, Awk, and Make Should Be Combined http://www.oilshell.org/blog/2016/11/13.html http://www.oilshell.org/blog/2016/11/13.html Make is just a small elaboration on the shell model (concurrent processes and files), but unfortunately it's implemented as a completely separate tool that shells out to shell! facepalm
- cjauvin 6y agoOne related thing that I began to do a while ago is to always have a `<project>.org` file going along any of my projects, which is a free-form Org-Mode journal of all the things and ideas related to that project, including of course esoteric commands for certain things (which sometimes can be hard to find in my bash history). With Org-Mode it's easy to fluidly change the structure of this journal, which makes it a very powerful tool.
- masklinn 6y agoSeems like it would make sense to make the file runnable, the "esoteric commands" would work as subcommands / makefile targets of sorts, and the rest of the orgfile would be what they aleady are.
- cyberbanjo 6y agoIn org-mode there's a concept of `tangle` and you can 'compile' (not sure if thats how org calls it) .org files into a number of individually specified scripts or documents. So you can have your top-level NOTES.org and also your scripts/* entangled. https://orgmode.org/manual/Extracting-Source-Code.html https://orgmode.org/manual/Extracting-Source-Code.html
- ijustlovemath 6y agoWhen I was working on a NASA project in university, one of my colleagues would always preach the virtues of org mode; as a person whose brain now thinks in vim, is it worth learning emacs just for this tool?
- asciimov 6y agoNot op, but yes org-mode is worth it. As a vim user, my recommendation is to skip spacemacs, and go for straight emacs (with evil mode if you like modal editing, i do) I liked Uncle Dave's Tutorial series on youtube: https://www.youtube.com/channel/UCDEtZ7AKmwS0_GNJog01D2g/videos https://www.youtube.com/channel/UCDEtZ7AKmwS0_GNJog01D2g/vid... Start with his emacs tutorial, it is mostly learn emacs and org mode as you learn to setup emacs.
- dreamer7 6y agoI have a similar practice too. For most projects I have *.cmd files such as start.cmd build.cmd run.cmd
- gglitch 6y agoSame, although I leave off the extension. https://www.talisman.org/~erlkonig/documents/commandname-extensions-considered-harmful/ https://www.talisman.org/~erlkonig/documents/commandname-ext...
- dheera 6y agoI have tons of such scripts for personal use, not just projects. Here's one I wrote a couple days ago called "jabra-stop-changing-volume-goddamnit": #!/bin/bash while sleep 0.1; do pacmd set-source-volume bluez_source.70_BF_92_CD_77_32.headset_head_unit 60000; done Also, lots of scripts related to ffmpeg and other tools where the command line arguments are too hard to remember. For example, "ffmpeg-extract-sound-from-all-files-in-dir": #!/bin/bash find *.mov | sed -e s/.mov// | xargs --replace=qq --verbose ffmpeg -i qq.mov -acodec pcm_s16le qq.wav
- ryanianian 6y agoSimilar notion: Never, ever have more than a simple shell script invocation in your CI configuration. So many projects I've worked on have had nontrivial build-logic or implicit assumptions in their CI configuration which leads to distrust of local reproducibility. Eliminate as many variables and steps to reproducibility as possible. At the very least, create a `ci.sh` script in your project and make all your CI invocations go through that. If you avoid putting "raw" npm or whatever invocations in your CI config in the first place you won't be tempted to add "just one more option" to your CI config and forget to add that option when running locally. Such a script also becomes a de-facto API for how to use your project. This can help immensely if all of a sudden, for example, you switch programming languages (or major versions) environment assumptions, etc.
- chubot 6y agoYes definitely, many lines of shell in YAML (or Docker files) is one of my pet peeves. It's always better to invoke a shell script. Then you can use ShellCheck on it, your editor will syntax highlight it, you can parse it with Oil (osh -n) [1], etc. And yes there shouldn't be too much of a difference between what developers do locally and what the CI does. It's sort of an anti-pattern if you have a bunch of automation that can only the CI can run. The developer should be able to run it on their machine too, without CI, and without YAML ... [1] http://www.oilshell.org/why.html#what-you-can-use-right-now http://www.oilshell.org/why.html#what-you-can-use-right-now
- paulryanrogers 6y agoThis is the ideal. In practice the CI environment may need optimizations that don't make sense locally without a lot more work.
- candiddevmike 6y agoI try to have CI only run make targets. Helps to reproduce things really easily.
- jolux 6y agoOften when you use the CI config instead of shell scripts it gives you deeper integration with the CI provider, like better error messages and such. I wish there was a way to bridge that gap.
- athenot 6y agoThese scripts become documentation. Not about what the commands are doing but WHY they are being ran. This in itself brings huge value.
- 3pt14159 6y agoWhat I do is make the same scripts from project to project then have helpful aliases to run them quickly. For example, since I do TDD and often work on a single test file for a while I have a script in every project called: bin/run Then when I want to run it I just type the letter r, since I've aliased r to run ./bin/run. It's super fast. To keep source control clean I add it to: .git/info/exclude Which allows my .gitignore to be the same as everyone elses. I used to used the global gitignore file, but I had issues at times.
- ed25519FUUU 6y ago> backup-db.sh Just a small nitpick. I’d like it if we collectively moved away from including file extensions for scripts. You never know when you want to rewrite it in python or do something else. Nothing more confusing than opening up “backup.sh” only to find it’s actually a ruby script and must be executed.
- dvdgsng 6y agoWe usually end up with implementing the same scripts in shell, cmd and Powerhell, since some Windows folks prefer not to install cygwin or use wsl. Its a PITA to maintain, but doable if the scripts are simple and only check for requirements and the actual work is done by python, groovy, go, whatever.
- thefifthsetpin 6y agoI handle that by linking src/backup.sh as bin/backup Then if I rewrite it as src/backup.py I just change the bin/backup link to point there instead.
- theptip 6y agoA follow-up recommendation I give (which I suspect might be unpopular with many around here) is to use Python for all but the most trivial one-liner scripts, instead of shell. Since 3.5 added `subprocess.run` (https://docs.python.org/3/library/subprocess.html#subprocess.run https://docs.python.org/3/library/subprocess.html#subprocess...) it's really easy to write CLI-style scripts in Python. In my experience most engineers don't have deep fluency with Unix tools, so as soon as you start doing things like `if` branches in shell, it gets hard for many to follow. The equivalent Python for a script is seldom harder to understand, and as soon as you start doing any nontrivial logic it is (in my experience) always easier to understand. For example: subprocess.run("exit 1", shell=True, check=True) Traceback (most recent call last): ... subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status Combine this with `docopt` and you can very quickly and easily write helptext/arg parsing wrappers for your scripts; e.g. """Create a backup from a Google Cloud SQL instance, storing the backup in a Storage bucket. Usage: db_backup.py INSTANCE """ if __name__ == '__main__': args = docopt.docopt(__doc__) make_backup(instance=args['INSTANCE']) Which to my eyes is much easier to grok than the equivalent bash for providing help text and requiring args. There's an argument to be made that "shell is more universal", but my claim here is that this is actually false, and simple Python is going to be more widely-understood and less error-prone these days.
- teknopaul 6y agofor the record... if __name__ == '__main__': args = docopt.docopt(__doc__) means nowt to me. In fact I don't really understand your post and I've written a lot of python. No language has a simpler api to executing a cli command than bash itself. By definition.
- divbzero 6y agoThe conditional expression __name___ == '__main__' simply tests whether this Python file is being executed from command line. It’s one of the ubiquitous Python idioms that doesn’t actually follow Python’s zen of “readability”.
- pbiggar 6y agoThis is exactly what we do for Dark: https://github.com/darklang/dark/tree/main/scripts https://github.com/darklang/dark/tree/main/scripts Not only that, but each script automatically runs itself in our docker container so it's fully repeatable. They all start with . ./scripts/support/assert-in-container "$0" "$@" which is just this: if [[ ! -v IN_DEV_CONTAINER ]]; then scripts/run-in-docker "${@}" exit $? fi And this is run-in-docker: https://github.com/darklang/dark/blob/main/scripts/run-in-docker https://github.com/darklang/dark/blob/main/scripts/run-in-do...
- Justsignedup 6y agoI spent a fairly sizeable time on this in one of my previous jobs. Most things were now bin/do_this.sh or bin/setup.sh or whatever. Setting up a pc went from 1-3 days to a 3 hour "run this setup.sh" script and call me if it crashes. It will become obvious when you hire someone that this needs to be done. As a project lead, schedule time for it!
- snake_case 6y agoIf you're not opposed to installing another tool for scripting, then check out my project mask [1]. It's a CLI task runner (made with rust, single binary) that supports multiple scripting languages and it's defined with a simple, human-readable markdown file :) I used to have a scripts directory for each project, but I really wanted basic argument/options parsing and subcommands. That's mostly why I made mask. Now I use it daily as a project-based task runner, as well as a global utility command. [1]: https://github.com/jakedeichert/mask https://github.com/jakedeichert/mask
- scns 6y agoThis looks really useful, thank you for making this.
- ultra_nick 6y agoWhy break tasks up into 10 line files when you could break them into 10 line Python/Similar functions? Same advantages as above, a more powerful cross platform language, cross-task references can be linted def run(params): "Run a built binary." subprocess.run("./" + params.path) . . . if __name__ == __main__: "Take CLI options and run selected function.""" # Parse CLI arguments # Call functions with options
- masukomi 6y agobecause it's not the unix way and the unix way has a LOT of value. because not all the tools you use in your system can import your python/similar file. Your deploy pipeline could involve 10 languages running under multiple os's / versions or to put it another way... because the whole world doesn't run in your favorite programming language.
- umaar 6y agoI've been moving most of my projects over to Makefiles, instead of npm scripts [1]. Unless I am missing something obvious, it's nicer to run: make vs. npm start make thing vs. npm run thing But also more importantly, the Makefile itself feels much cleaner, rather than stuffing scripts in package.json files. The thing I'm confused about, why do npm scripts seem to dominate? I don't see many people using Makefiles. [1] https://github.com/umaar/video-everyday/blob/master/Makefile https://github.com/umaar/video-everyday/blob/master/Makefile
- mstade 6y agoI had your point of view until I started working in corporate environments where windows reign supreme, and then I understood that the reason why no one wrote Makefiles or shell scripts is simply because no one had access to those runtime environments. JavaScript, however, truly is everywhere these days. Sure, you can run Make and even shells on windows now, but that doesn't mean you have the power or even capability to do so in a corporate environment where IT dictates only reluctantly provides you with a machine in the first place.
- stouset 6y agoLife is too short to work in these kinds of environments, if you can avoid it.
- mark_and_sweep 6y agoI actually find using Make on Windows to be really simple. I usually drop a prebuilt make.exe into the project directory since it's just 244kb (http://www.equation.com/servlet/equation.cmd?fa=make http://www.equation.com/servlet/equation.cmd?fa=make, https://github.com/MarkTiedemann/make-for-windows https://github.com/MarkTiedemann/make-for-windows). If the Makefile needs to be cross-platform, that's easy to do, too: ``` ifeq ($(OS),Windows_NT) # Windows else # *nix endif ```
- lmm 6y agoMakefiles are much less cross-platform and much less clear about their dependencies. A hand-written Makefile probably won't work on FreeBSD unless you've deliberately tested it there, much less Windows, and is probably relying on a bunch of GNU-only flags in the utilities it calls. Some of them are probably from a newer version of that utility than what ships on OSX. Etc.
- maest 6y agoSounds like the main function here is to create concepts at the correct level of abstraction, a bit like an API.
- austincheney 6y agoMy experience with simplicity is that it requires greater effort. Simple is not easy. As such, people dedicated to the task of effort avoidance will find a way to avoid increased simplicity. The justifications and qualifiers are creative and elaborate often themselves taking great effort, often greater effort than that they originally sought to avoid. I see this behavior repeated so frequently in software and with such profound conviction. A brief example is a single method call from a language or platform supplied API used to solve a problem instead of the entirety of a large framework. The heresy of such a disgusting travesty.