7 ms·
Ask HN: Any tool for managing large and variable command lines?
I often find myself building up big complex command lines for testing my code, which live in my shell history.
Sometimes it's possible to just wrap these up in scripts but often my invocations of that script get so variable that I need many flags etc and now the commandline gets really long and I'm back at square one.
A little hard to describe in the abstract so here's a present example:
- I wanna compile a kernel and boot it up in QEMU, then SSH into it.
- But sometimes, I wanna skip the build (it takes a few seconds to detect that the tree is unchanged)
- Sometimes, I wanna boot with a different special disk file.
- Sometimes, I wanna change the networking setup. Sometimes this requires me to charge the part of the commandline that SHH's into the VM, to match the changes I made to the QEMU commandline.
- etc etc
So I just have a huge command line in my shell history and I bring it up and manually edit it each time.
But I think it's possible to make this process more convenient. An inspiration for this is Magit, an Emacs plugin that allows (among many other things) building up and stringing together lots of complex git commandlines with varying flags, via a few keypresses in a discoverable text UI.
I wonder if something like this already exists? If not I think it could be a fun design challenge. By looking at the shell history and using heuristics about common CLI design patterns perhaps you could automatically create keyboard shortcuts.
- austin-cheney 2y agoI would do that via a custom Node application executing commands as child processes driving a container.
- nequo 2y agoIf you're using bash, you can edit long commands by hitting C-x C-e. This will open an editor with the command in the buffer. When you save and exit, bash will execute the edited command. This is not in the style of `transient` in Emacs but it could work well, especially with your wrapper script that takes command line flags.
- jmholla 2y agoIn addition, I think bash's `operate-and-get-next` can be very helpful. When you go back through your shell history, you can hit Ctrl+o instead of enter and it will execute the command then put the next one in your history on the command line, and keep track of where you are in your history. This way, you can rerun a bunch of commands by going to the first one and Ctrl+o till you are done. And you can edit those commands and hit Ctrl+o and still go to the next previously run command. Note: fzf's history search feature breaks this. https://github.com/junegunn/fzf/issues/2399 https://github.com/junegunn/fzf/issues/2399 Edit: And up and down will still keep the relative position you are at in your history in case you want to skip a command.
- jonS90 2y agoI use this in combination with FZF's ctrl-r command history which allows me to fuzzy search command history.
- x11antiek 2y agoI am a simpleton, me using CLI is basic compared to what you've described, but I have a tool called Warp on my Mac which helps me with saving terminal commands and making notes on what/why they do stuff. It's free and it's cool imo.
- philomath_mn 2y agoI started using just [0] on my projects and have been very happy so far. It is very similar to make but focused on commands rather than build outputs. Define your recipes and then you can compose them as needed. [0] https://github.com/casey/just https://github.com/casey/just
- jarpineh 2y agoI second this. Though I might just (heh) be too biased having today added some common Just recipes as global shell aliases [1] I also use Just for build and deploy commands. Other thing I was thinking of was setting environment variables. Complex pipelines can be split into several dependent recipes. Or using shell aliases you can string them together yourself with regular pipes in very compact form. Just recipes defer to shell or a programming language of your choice. Just makes some shell stuff easier without replacing or hiding it. [1] https://just.systems/man/en/chapter_68.html https://just.systems/man/en/chapter_68.html Edit: I also added Atuin https://atuin.sh/ https://atuin.sh/ recently to better deal with shell command history. Though that one I’m still learning.
- martinbaun 2y agoI have a `run` bash file where I just return exit 0 on third line. Then I can copy/paste anything I need up from below the exit 0. Silly, yes. but it's super easy and extremely fast.
- secwang 2y agoIt's great.
- martinbaun 2y agoThanks. Simple and dirty, but works everytime. Even more, I put a vim shortcut so I can run it without leaving the terminal.
- quesera 2y ago> exit 0 If you use `exit $?` instead, you will preserve the exit status from the previous command.
- martinbaun 2y agothanks! Learning every day
- nailer 2y agoAfter about 30 years I'm really good at shell. But I don't think it's the best for managing complex workflows. You could use a local port of something like github actions, but personally I'd pick a language (Python, TypeScript, Ruby) and make a function that takes an input kernel and optional disk and networking config, builds if the tree has changed, and returns the ssh command to the created VM.
- sargstuff 2y agoYeah, but that's doens't make for an april 1st way to get arrays.[1] [1] : https://unix.stackexchange.com/questions/344360/collect-exit-codes-of-parallel-background-processes-sub-shells https://unix.stackexchange.com/questions/344360/collect-exit...
- nailer 2y agoI understand the link, but not your comment.
- sargstuff 2y agooverly complicated writing using just shell vs. programming language(s) designed for/more suited for task, snarky: get a raise via get_arrays approach works nice if billing by hour, not so much if scripting system wide scripts.[0] [0] : https://www.linux.org/threads/init-and-its-alternatives.9279/ https://www.linux.org/threads/init-and-its-alternatives.9279...
- nepthar 2y agoI made something called shell workspaces[0] for this. I like to tinker with bash and this has helped me keep all of the commands relevant to a particular project discoverable, accessible, and documented. It uses a function "," for command execution. So, for instance, if the workspace file defines a function "build", when you're in the workspace, the command `$ , build` will run it. It's not the most comprehensive solution to this, but it's probably the bit of shell programming that I use almost every day and has saved me tons of time. [0] https://github.com/nepthar/ws.sh https://github.com/nepthar/ws.sh
- Jeff_Brown 2y agoI'm not sure I understand. Is the idea to step through a script, maybe executing the default value of each line, but offering the opportunity to edit that line before executing it?
- akincisor 2y agoI've recently used a ruby script with a bunch of methods that I can configure with a few yaml files to generate a shell script which I can then run. Each run generates a timestamped folder with not only the bash script, but also a copy of the configs used to generate it, and all the data that the commands in the bash script needs (json files). I find this style of generator a common pattern for commonly used but frequently tweaked scripts.
- Kalanos 2y agorather than invoking the script with arguments, you could just define the variables at the top of the script and run it
- Jeff_Brown 2y ago+1 Unless you depend on the results of partial execution to decide how to execute the rest of the script, this seems like the most natural solution.
- dusted 2y agoI have removed limit for bash history lines and file size and am using https://github.com/junegunn/fzf https://github.com/junegunn/fzf for reverse-search, and of course I use ctrl+x ctrl+e to edit long command lines in vim (I have export EDITOR=vim in bashrc)
- OJFord 2y agoSounds like you want `just` (pretty much a riff on make, but more designed for this use case). I don't find much use for it personally, but I think it fits what you're looking for.
- kazinator 2y agoFor that specific example, I might think about a script which asks interactive questions. Do you want to do the kernel build? [Y/n] _ Which diskfile to use: /path/to/diskfile_ # pre-stuffed editable input Or else well-documented command line options, nicely explained with --help.
- gverrilla 2y agoI'm an apprentice programmer. I also like interactive programs. I use the library questionary[0] in Python, and ChatGPT is great at providing the necessary code in no time. 0: https://pypi.org/project/questionary/ https://pypi.org/project/questionary/
- lambdaba 2y agoI use vi-mode and when i need to make more elaborate edits, I have this binding in zsh to edit the current line in $EDITOR: autoload -Uz edit-command-line zle -N edit-command-line bindkey -M vicmd v edit-command-line For "normal" Emacs keybinds this is the standard: bindkey '^x^e' edit-command-line As others have mentioned this goes well with fzf and its shell history plugin. But, in truth, the shell interface could use a major overhaul to make it more like a notebook, there are early attempts at this (e.g. Warp), but it's still quite timid IMO. I'll also say this is one of the perks of being a vi user, it's the CLI usage where it really shines, I can make surgical edits really quickly, thanks to all the jump and change keybinds that are so central to vi usage.
- seivan 2y agoSnipkit and Pet. I prefer Pet, mostly because Snipkit makes me verify 20 times (slight exaggerated) and doesn't give me the actual command so it'll be part of the shell history. Honestly, just dump the command in my shell and let me press enter and make that the confirmation, but whatever. You can give arguments with default values, so they are reusable across projects.
- nikolay 2y agoI highly recommend Run [0]. [0]: https://github.com/TekWizely/run https://github.com/TekWizely/run
- izoow 2y agoI rely on an unlimited bash history and fzf to fuzzy search through it. Commands that get too long and complicated might get thrown into a simple bash script, often quite literally "thrown" as a single line without any cleanup or extra logic. I've tried making pretty scripts with many options, but as you said, then you just end up with another tool with many complicated options, not to mention the time it takes. I noticed that the quick and dirty scripts that I often just copy paste and tweak if I want to get a different functionality get by far the most use and survive the longest. I've also been meaning to have a look at invoke[1] which seemed interesting, but I haven't gotten to it yet. [1] https://www.pyinvoke.org/ https://www.pyinvoke.org/
- apimade 2y agoGUI.
- dogline 2y agoIf I understand you correctly, this is where I just use a Makefile, but keep it very simple ```render:\tffmtpeg ...``` Do a have dozen of those, and you now have a `make render` command to run that task.
- binarymax 2y agoWhich shell? If you’re using bash, you’re missing out - so I’d first upgrade to zsh or fish, which have better options. The other thing I’d do is get really really good at writing shell scripts. Also nowadays it’s super easy to take an existing complex one liner and ask GPT to make it a robust script with args (but check it of course!) I also like some of the other suggestions if you’re predisposed to them already, like Makefiles, which are simple, flexible, and powerful.
- teyc 2y agoI’ve used this https://github.com/secretGeek/ok-bash https://github.com/secretGeek/ok-bash It is a menu that works off a file that contains a collection of single line commands. I like it because it is a local command that is only relevant to the project I’m working on, and sometimes after months of absence it is easy to pick up where I left off.
- deleted 2y ago[deleted]
- xlii 2y agoLately I’ve been running such shell scripts/small commands through org-babel in Emacs. They can be fed one through another which is cool. Also can recommend Justfile (it has great discoverability and it just works).
- moritonal 2y agoA collection of Jupyter notebooks with "!" calls to binaries. The python requirement is inconvenient but the consistency, output tracking and variables make it worth while.
- philipov 2y agoPut all the command-line configuration into a shell script with zero arguments.
- bugbuddy 2y agoA script file?
- prima-facie 2y agoI've heard good things about atuin https://github.com/atuinsh/atuin https://github.com/atuinsh/atuin
- codetrotter 2y agoI use it, it’s pretty good. For some reason it seems to try to connect to some atuin server even though I am using default config. That worries me slightly sometimes. I also had one case where I was using ssh from my home to a machine at one of my parents place and there was an ongoing packet loss problem and so every time I press Ctrl-R to run some previous command I had to wait actual ages for the terminal to redraw because it opens atuin and shows as many lines of history as it can and also updates when you type to search. Of course that is an extreme situation, but it was wild nonetheless to see how much worse the command line experience became in those conditions because of that. I still use it though, and will continue to as well.
- tonymet 2y ago1. .sh scripts 2. makefiles 3. If you're using bash, try the edit-and-execute-command command. By default, this is assigned to Ctrl-x Ctrl-e (type ctrl-x, then ctrl-e).
- kjkjadksj 2y agoI would just have a script with either all the variables on top of the script for you to set, or have it look for them in a configuration file you might change a little every run. That’s just my preference for this sort of thing, though. If I can avoid using external stuff or spending time learning new syntax, and keep it as simple and stupid as possible I try and just do that. Sounds like your solution is simple enough with shell history file keeping a record for you. Maybe playing with aliases or writing functions in your bashrc could save some typing for you
- invalidator 2y agoI create wrapper scripts. I try to keep them simple... they're convenience scripts, not a replacement for a build system. Here's one for a QEMU dev env where I add and remove bits of command line in a freeform way. I just break out the command line into components, and uncomment the ones I want for a particular use: #!/bin/bash set -ex # Drive created with: # /usr/bin/qemu-img create -f qcow2 ngx.qcow2 40G #vga='-vga cirrus' vga='-vga std' #vga='-vga qxl' drive='-drive file=ngx.qcow2,cache=writeback' tablet='-usbdevice tablet' #vnc='-vnc :3' #cdrom="-cdrom base_install.iso" #cdrom='-cdrom documentation.iso' #cdrom="-cdrom /dev/sr0" cdrom='' #net='-net user -net nic' net='-net user,restrict=off,hostfwd=tcp::6666-:6666 -net nic,model=rtl8139,macaddr=00:11:82:C3:54:55' #net='-net user,restrict=on,hostfwd=tcp::6666-:6666,hostfwd=tcp::3389-:3389,hostfwd=udp::3389-:3389 -net nic,model=rtl8139,macaddr=00:11:82:C3:54:55' monitor='-monitor stdio' qemu-system-x86_64 -enable-kvm -nodefaults -m 768 $vga $drive $tablet $vnc $cdrom $net $monitor If you have multiple stages to your build/run process, you can chain together a few scripts. build-and-run.sh builds it, and then calls run.sh which runs it. If you want to skip the build step, you use run.sh.
- Dbuild 2y agoI created a tool I called parseArger to create/maintain bash scripts arguments/options/flags parsing code https://github.com/DimitriGilbert/parseArger https://github.com/DimitriGilbert/parseArger It's heavily inspired by ArgBash (https://argbash.dev/ https://argbash.dev/) and uses parts of bashly (https://github.com/DannyBen/bashly/ https://github.com/DannyBen/bashly/) which are very similar tools :) I hope that helps and please forgive the shameless plug...
- untech 2y agoCan’t you use aliases or functions in the ~/.bashrc? That’s what I do for complex lines I want to reuse.
- jedberg 2y agohistory | sed 's/^ *[0-9]* *//' > commands.sh vi commands.sh (edit the file to only include your big nasty command lines, then comment them all out) chmod +x commands.sh Now every time you want to run a command, you just uncomment that line. If it needs changes, you can copy the line, change it, and now you have another command. You can even add comments to help you remember what each command is for.
- bomewish 2y agoWow this is kinda elegant. An improvement I can think of is putting it on your PATH so you can just invoke “commands” directly each time?
- dgfitz 2y agoCompose all your commands into “does one thing well” and abstract the “thing” it does to an argument… if you have to. Then just chain them together as needed.
- replwoacause 2y agoI save stuff into SnippetsLab (macOS only)
- kbknapp 2y agoNavi[1] is perfect for this! It's both a fuzzy finder of a personal collection of commands, but it's template syntax is flexible enough to be able to have "command builders" I have a blog on doing exactly this for a subset of strace[2]. 1: https://github.com/denisidoro/navi https://github.com/denisidoro/navi 2: https://kbknapp.dev/strace-navi/ https://kbknapp.dev/strace-navi/
- jauntywundrkind 2y agoI'm going to recreate something I did a while back, which was to create a vim command/script which would: copy the current selection or line to a specified term window, and hit enter. And then version two, do that, but then copy the contents of the line/selection into a new line. Basically using vim as an input buffer. With the option to copy the current line- when working systems there was often incremental improvement that I was trying to do, so running then creating a new line was valuable. I probably should just learn how to insert a line offset into the current line though. What's still missing in both my vim-driven world and in shells is any ability to make references & changes. Ideally I'd like to be able to have lines that interpolate vim registers & more complex expressions. Ideally I want a shell that lets me see my previous raw uninterpolated expressions, that hps me distil out pieces for reuse. The way that history just shows the interpolated version destroys the ability to see what & where I'm borrowing bits of the past that I could be using as hints to re-compose new futures.
- from-nibly 2y agoLearn bash. Make a bash file. Make it executable. Put it on your path, or create a ~/bin directory and add that to your path. You can write whatever crazy nonsense you need.
- BerislavLopac 2y agoIf you're comfortable with Python, Invoke [0] can be a great help. You can include your bash code directly inside it, or collect your scripts in a directory and run them from Invoke tasks. Plus, you can easily use all the power of Python on top of your scripts. [0] https://www.pyinvoke.org/ https://www.pyinvoke.org/
- sargstuff 2y agocopy/write edited shell history to unique shell file for each set of edits. make shell file executable. run shell file containing action want to use.
- laktak 2y agoSee https://github.com/laktak/tome https://github.com/laktak/tome It lets you manage playbooks, interactive scripts that keep your recurring commands (or what you would otherwise search for in the bash history).