30 ms·
Show HN: Ultimate Plumber – a tool for writing Linux pipes with live preview
- ptman 8y agoCareful with rm: r rm rm - rm -rf rm -rf ~ rm -rf ~/ rm -rf ~/tmp # where did my files go?
- akavel 8y agoRight, it's theoretically dangerous, or I'd say practically: "please think when using it"... though I personally couldn't find any other serious problematic example other than typing something clearly dangerous in the first place. And I personally found it immensely useful in the other, non-dangerous cases. So I just took the chance and went with the idea, and then wanted to share it with you, so you might also find it useful, maybe! :) Also... umm... you know, a tool isn't really truly Unixy if you can't hurt yourself with it, right? ;P That said, I'm starting to think — should I include a note about the potential dangers in the readme?... they seemed glaringly obvious to me (at least in case of rm-like commands), but can there be readers/users, to whom they'd not be obvious?... edit: Ok, I tried to add a warning in the Readme, that's something I can do easily, and if it helps someone, worth it...
- skybrian 8y agoIt seems like a whitelist of safe commands would be a good idea? The user can override with -f if they want to live dangerously. Even rm has a -f flag to override some safety measures, though the defaults aren't very safe.
- drb91 8y agoEven better, put it into a non-privileged user by default with no write permission.
- akavel 8y agoSounds interesting; but is there any way I could change to a different user if not running as root?... also, even if yes, how do I dynamically create a non-privileged user, or find a pre-existing one?
- gwn7 8y agoI would try creating a non-privileged user "up" during installation and switch to that user under the hood before running the command via `su`. (Though it would complicate installation which is effectively non-existent since there is a single binary; and tightly couple the application with the environment. It's a trade-off anyway.) (Edit: Somebody mentioned the user "nobody" which seems to be a better alternative.) In any case I think that the whitelist approach would work better as it would be nice if the tool knew in advance which commands work properly and which ones don't. That way it could inform its users about the allowed & unallowed commands. Commands with side effects (or "impure" in the FP terminology) don't make much sense anyway to be used within this. The main value is fast iteration, in order to verify the expected output of the pipeline. For me it doesn't make much sense to use it with commands whose main purpose is to modify the filesystem instead of generating / transforming data and writing it to stdout. So the criteria for the commands that would make it into the whitelist may be "not having side effects", "writing to stdout", and optionally "reading from stdin". Fastly iterating side effects is unsafe as well, as has already been pointed out. This can be a fine data analysis tool actually, unfortunately only for command line geeks. The experience is actually not so far from analyzing data with VIM interactively by piping it to an external UNIX command and getting the result dataset back into the editor. VIM hackers will know :)
- seiferteric 8y agoI actually like the user idea better. The problem with a whitelist is that even useful commands can have subtle or little known dangerous modes, like "find . -exec rm {}"
- gwn7 8y ago
- CGamesPlay 8y agoNice tool! Your main use case for destroying files is “xargs rm”, which would be used when receiving a piped argument list. Could you have this command run as user “nobody” by default, and have a flag to run as a real user? Or as others have mentioned, use capabilities on the spawned processes to prevent anything other than standard input/output?
- etatoby 8y agoI'm not so versant about the state of permission / capability systems on Linux. But if there is a way for you to drop the capability to modify the filesystem before running the first pipeline, you should definitely do it by default (and provide a switch to override it if somebody knows what they are doing.)
- somanyquestions 8y agoyikes! It's always been surprising to me that their isn't a built in "undo" to `rm` e.g. why doesn't `rm` just move files to /tmp?
- clarry 8y ago> e.g. why doesn't `rm` just move files to /tmp? And do what when tmp gets full and services & applications start crashing? When someone implements that undo, someone's going to need to write a tool that really does remove files, really. Will the next guy wonder why really-rm doesn't have undo built in?
- diegoperini 8y agoWhat about undo that expires in 1 minutes? Gmail website does that for sent emails (for 5 seconds I guess).
- hk__2 8y ago> What about undo that expires in 1 minutes? Gmail website does that for sent emails (for 5 seconds I guess). Same question as parent: "And do what when tmp gets full and services & applications start crashing?"
- kungtotte 8y agoYou could do what is more or less an industry standard when it comes to this and warn the user that there's not enough room to allow an undo when deleting said file(s) and prompt if they want to delete it permanently instead. It's not rocket surgery.
- jxy 8y agoI'm never that nervous when doing `rm -fr` on my Plan9 or FreeBSD or macOS systems, which all have frequent snapshots turned on. We could all just work with WORM drives instead.
- hinkley 8y agoI found a new variation recently. Luckily all my code was checked in. Meant to type: rm -rf node_modules && npm install Question: what's immediately next to the & key on a keyboard? Answer: * Actually typed rm -rf node_modules ** npm install Oops.
- nothrabannosir 8y agoIf you’re on Mac OS X , try installing the “trash” app (from brew iirc) and retrain yourself to never use bare rm anymore. There’s probably similar tools on Linuxes. Theory is: rm is dangerous and should not be used day to day.
- mynewtb 8y agoOr simply have local, quick, regular and tested backups.
- dzamo_norton 8y agoUsing btrfs on Linux I have a job that takes a CoW snapshot of the whole filesystem every 10 minutes just so that I can undo whoopsies. I don't retain all of these snapshots, only the most recent one.
- ken 8y agoThis is one of the main reasons that I ultimately decided for my Strukt.app to not be built on top of built-in shell commands. For an interactive tool, it's crucial for each operation to be able to report whether it's idempotent, does writeback, etc. Strukt doesn't do 'rm' yet (and nobody has ever requested it), but it does have other operations that do writeback. Once you add one, automatic updates are disabled, and you have to explicitly run the pipeline each time you want it run. We really need a new cross-platform low-level framework for lego-like operations, like shell commands but updated for the 21st century.
- jodrellblank 8y agoPowerShell has a design answer for this - each cmdlet takes parameters, but the environment provides common parameters relating to loging, error handling, storing output in variables, and it provides support for "WhatIf" and transactions. WhatIf instructs the command not to make any changes, but merely to log what it would do. You can "Remove-Item C:\Windows -Force -WhatIf" and it will output "What if: Performing the operation "Remove Directory" on target "C:\windows"" but not delete anything. (implementing support for "whatif" is optional, so you can't casually use it on any given cmdlet without first checking that cmdlet supports it and makes sensible use of the parameter instead of ignoring it, but the idea is there).
- ken 8y agoYep, that's basically the solution that I came up with, too. I have a "dry-run?" flag planned, if I ever end up with enough writeback operations where it'd be useful. (That's the common Unix shell name for it, though it's not entirely consistent -- which is another problem with using Unix built-ins!) One thing I've noticed is that it's really frustrating (and not at all helpful) to require perfect validity from a shell-like tool. There's just too many times where you want to edit something into something else, without needing to find a perfect path of valid and meaningful syntax at every intermediate step. So if you have a flag on an operation that doesn't understand that flag, the flag will have a (?) badge to indicate that it won't be understood or used, but the pipeline will still run.
- 8y ago
- djsumdog 8y agoThat's the first thing I thought and it's mentioned in the readme file itself. I don't think I'd ever want to use this tool. Every keystroke could be an arbitrary command, and I wonder about the performance of commands with a lot of I/O or processing. Conceptually it's really neat and I get it, but practically I find things like parameter completion in Fish much more useful.
- deleted 8y ago[deleted]
- miduil 8y agoJust an awesome tool, I love those moments of "ah that's trivial, why didn't I think of this already" and it definitively applies for this tool. So special kudos for the idea, but also how little go code the implementation took (ignoring dependencies). There is already some "pre-release" discussion going on over at lobsters btw. https://lobste.rs/s/acpz00/up_tool_for_writing_linux_pipes_with https://lobste.rs/s/acpz00/up_tool_for_writing_linux_pipes_w...
- akavel 8y agoThanks! :) See also my "prior art" note in the readme ;) I also had the "why didn't somebody think of this already" moment when I thought of it ;) And somebody in fact more or less did, just I wasn't able find it then :)
- mohammedbin 8y agoWhy wasn't there a tool for this earlier? I am a very aggressive piper but I used to hit up arrow , a series of ctrl-w and enter and wait everytime. Thank you
- esotericn 8y agoThis is awesome, but I am absolutely bloody terrified of using it on my systems. As is stated in the README, if you write 'rm' or anything like it... oopsie oops.
- nickspacek 8y agoHmm, is there a straightforward way to be a bit more careful without detracting from the usefulness? What about a default blacklist of commands with the ability to override it through a config file?
- rcthompson 8y agoI don't think a blacklist could possibly be comprehensive enough. I think you'd have to use some OS permission-limiting system to prevent it and any subprocesses it spawns from have any write access to the filesystem.
- esotericn 8y agoPretty much. Is rm blacklisted? OK. How about bash -c "rm"? cp? mv? vim? ...? :D
- y4mi 8y agoYah, a whitelist of commands which includes bash would probably be best. You'd be fine using it and can simply switch to chainsaw mode by adding bash to the command
- malkarouri 8y agoI think the tool would be much more useful with a whitelist. Do this only for grep, awk, sed and other similar tools. Of course, much more thought is needed to try something like this. Somebody could as well use awk with its system command to do whatever..
- deleted 8y ago
- traviscj 8y agoNicely done! I do a pretty similar version of this with command-| in textmate, but I've wished for/thought about building a command like this to speed up the process! The textmate approach does have the advantage (or perhaps disadvantage, for large enough inputs!) of emitting immutable "views" of the data at each step, rather than (presumably?) re-evaluating the pipeline each time, which is nice if the steps of the pipeline are the slow part, or to go back 2 steps and start a new pipeline. Maybe a potential future flag? :-) Automatically emitting an `up<N>.sh` script is super clever too!
- akavel 8y agoCan you please elaborate a bit on the textmate approach? I don't fully grasp it from this description yet... while it totally sounds interesting...
- traviscj 8y agoI (partially) recreated your `seq 1 15` example here [1] -- no sleep, but my lshw wasn't as interesting as yours :-) I'm hitting command-shift-| to bring up the "filter through command" dialog. [1]: https://tcj.io/i/textmate_piping.gif https://tcj.io/i/textmate_piping.gif
- akavel 8y agoOk, I get it now. Thanks! :)
- em-bee 8y agoi have not used textmate, but the description let me come up with this idea. maybe it's useful: consider the command A | B | C | D when you process it in up, split the command at the | and process each one separately. that is first run A; save the output into a buffer (BA) then pipe BA into B, save that into BB, and so on, until you have 4 buffers containing the outputs of A; A|B; A|B|C; and A|B|C|D; now when the user edits, you can see which section is changed. say the user changes the command C into C1, so the pipeline becomes: A|B|C1|D at this point you can pipe buffer BB (which has the output from A|B) into C1; write that into a new buffer BC1; and pipe that into D; the old buffers C and D can be removed at this point, or you can keep them around in case the user undoes their change and goes back to an old version. greetings, eMBee.
- codetrotter 8y agoWell damn! This is a really elegant solution for a problem that has been bothering me for quite a while, and for which my own planned solution was much more complex :O
- akavel 8y agoThanks a lot for the good words! :D <3
- jlv2 8y agoOne of my favorite quotes: ``Remember, a temp file is just a pipe with an attitude and a strong will to live'' - Randal Schwartz [comp.unix.questions]
- jlv2 8y agoOne of my favorite quotes: ``Remember, a temp file is just a pipe with an attitude and a strong will to live'' - Randal Schwartz [comp.unix.questions]
- pjungwir 8y agoThis looks like a fun and helpful tool! (Perhaps best to ignore the notice from lshw and not run it as root, ha ha. Maybe even only in some kind of sandbox. I bet they could add a --safe flag so it would make an on-the-fly Docker image with a copy of your cwd, and run the commands in there. Then you have different problems I suppose, but still....) I've never seen `paste - -` before, but that is amazing too! I can't count how many times I've wanted something like that. Surely that will help my own shell pipelines in the future. Also I've never seen `foo |& bar` before. What a neat idea! The surprisingly-valid creativity of it reminds me of Duff's Device. Has anyone else ever found uses for that idiom? After so many years working in the shell, it's such a rare and delightful treat to learn new tricks. Thank you for sharing this with us!
- ktm5j 8y agoMaking a docker container to run the command in is a good idea, but not helpful if the container doesn't have the same filesystem. I love the idea of something like this, but I'm not sure if I would ever use it out of fear of unintended actions.
- DanielRibeiro 8y agoIt is a pretty neat trick. Kinda hard to Google for though. Here is the bash manual page on it: https://www.gnu.org/software/bash/manual/html_node/Pipelines.html#Pipelines https://www.gnu.org/software/bash/manual/html_node/Pipelines...
- lloeki 8y agoIf you didn't know about paste(1), maybe you don't know about comm(1) either. There are a couple other like that that I typically fail to remember because I use them only every now and then but when I do they usually are a lifesaver.
- the8472 8y agoThis is nice for linear pipes. But occasionally I find myself building more of a graph. Bash lets you provide file arguments from pipes via the <() syntax (it's turned into a fd reference via proc behind the scenes). And you can also wire up additional pipes with fd renumbering, though this gets ugly in bash, some other shells are more flexible there, e.g dgsh. A tool to iteratively build a graph of commands and pipes would be the next step.
- mohammedbin 8y agoThis sounds interesting. Can you pull up some history and make a gist/pastie?
- akavel 8y agoThat's totally where my long-term vision goes! :D But I'm very unsure if I'll have enough time and resources to be able to reach it :/ My strong hope is however that the Luna language[1] & team may get us there faster, and to an even nicer place actually :) [1]: https://luna-lang.org https://luna-lang.org
- jspears 8y agoWhat about using LD_PRELOAD to load a FS shim that prevents writing to the FS?
- _kst_ 8y agoWhat about making the tool behave sanely so you don't have to play tricks with the OS to protect yourself from its behavior? If I type "grep", I don't want it to try to execute "g", "gr", and "gre" as I type it -- not if those commands don't exist, and especially not if they do. Just don't execute a command unless the user requests it (say, by typing Enter). Problem solved.
- jspears 8y agoIt would remove the live execution bits which are kinda neet. I think the cli has a lot of life left in it and ideas like this are worth exploring. Perhaps compromises are necessary but I would love a tool that pretends to execute something shows the results and gives the option of committing those changes. Think of a find | xargs grep | rm That then shows the output but prompts the user for commit.
- akavel 8y agoI don't think LD_PRELOAD would be enough; there are programs which don't use libc and reach straight for syscalls (e.g. the whole Go ecosystem/runtime)
- jspears 8y agoSure; it wouldn’t be perfect safety but would work for a large swath of Unix tools.
- sisk 8y agoThis is great. Thanks for sharing. For something a lot dumber: here's a tiny thing to print out pipe contents to allow for cancelation before continuing the pipe. https://gist.github.com/jasisk/be34ae93b74a3d3e8f0a4daac1237966 https://gist.github.com/jasisk/be34ae93b74a3d3e8f0a4daac1237...
- JdeBP 8y ago* https://github.com/akavel/up/blob/master/up.go#L574 https://github.com/akavel/up/blob/master/up.go#L574 Please at least respect the SHELL environment variable. (-:
- akavel 8y agoEheh; sorry, MVP! ;) and thanks a lot for taking it lightly and commenting in such a kind and playful way :D It's totally on the long list in some way... please see https://github.com/akavel/up/issues/2 https://github.com/akavel/up/issues/2 ! :)
- undershirt 8y agoI checked out the original tool Pipecut—it allows you to look at the output after each pipe: https://youtu.be/CZHEZHK4jRc?t=1273 https://youtu.be/CZHEZHK4jRc?t=1273 When I see stuff like this, I check the date to see if it happened before or after "Inventing on Principle" was presented in 2012. (It's always after)
- akavel 8y agoHmmm, you made me think and try to recollect. And to my surprise, I can't really remember the fact of The Presentation being an influence. But I am quite sure it must have been, I can see no other option. Hm, memory is a tricky thing. And Bret Victor left his imprint on our whole industry.
- gregwebs 8y agoThe most frequent use case for this for me would be interactive grep. The "fzf" tools has this along with fuzzy search. And the text doesn't disappear while you type "grep" like it does with "up". I will definitely try this out too though when I make a pipeline with sed, cut, etc.
- isodude 8y agoYou know that less has interactive grep right? Type & phrase <enter> To go back: & <enter>
- dustfinger 8y agorm is not the only issue. Another problem may arise if your command is not idempotent with respect to the state of the system. Especially if it is not idempotent with respect to system state and requires multiple parameters, since it will execute many times as the parameters are typed. Maybe having a configurable debounce on key press with a reasonable default would be sufficient to get around this issue. What would be really cool is if this tool ran in a pretend mode and showed you what the results would be without actually mutating the system. Then, once the user is happy with the results, the command can be executed for real. This whole concept reminds me of helm for emacs :-). Thanks for sharing!
- akavel 8y agoThanks for the ideas, and for the good words! :) Do you know of any nice screencast/video where some similar functionality of helm would be shown? Does it also work for shell commands in some way? I'm not an emacs user...
- swiley 8y agoMaybe if it automatically chrooted everything to prevent side affects (or better yet: isolate them to a ramdisk.) That would be awesome.
- amarshall 8y agoReminds me of fzz [0], which is a similar tool but focused more on interactive modification of prescribed components of a known command/pipeline than appending a pipeline. [0] https://github.com/mrnugget/fzz https://github.com/mrnugget/fzz
- _kst_ 8y agoI've submitted an issue on GitHub, title "Executing incomplete commands is too dangerous". https://github.com/akavel/up/issues/8 https://github.com/akavel/up/issues/8
- jxy 8y agoI do progressive filtering a lot. It's convenient to have all the intermediate results somewhere temporarily. What I do is typing the command in one acme window with the working directory, and middle mouse swipe it. Acme shows the result of that command in a +Errors window with the same working directory. I change the name of this new popped up window (to something like `+Errors-cmd`, type a command that operates on the file name `/mnt/acme/$winid/body`, and middle mouse swipe it again. Acme shows the result of the new command under `+Errors` again. Rinse and repeat until I'm satisfied. I guess this way it gives a convenient temporary place for all the intermediate texts. I can of course edit them in place if I want.
- d0mine 8y agoRelated: percol utility allows to narrow a process output interactively https://github.com/mooz/percol https://github.com/mooz/percol I use it all the time to select a command from my shell history. It is a very convenient addition to zsh-autosuggestions (one of those things that you use hundreds times per day without even thinking about it).
- ddalex 8y agoLate to the party. This screams like it needs to be part of the shell instead of an addition. E. G. I type a command in the shell, end it with pipe, and press enter, it doesn't delete the command but runs a preview of the output top 10 lines? This way I can continue to the edit the command without losing the context, the pipe, or I can decide to redirect it to a file!
- akavel 8y agoHmm, interesting thought! The tool needs to do some ugly buffering however; would there be place for it in a shell? Also, I suppose some kind of "Ctrl-Enter" or something would be a better idea instead? So that it could also be hit at any point in the line... But how about scrolling through the preview?
- nixpulvis 8y agoI've been considering how I might implement this as part of "Pipe old commands without rerunning" from my project's README. https://github.com/nixpulvis/oursh#features https://github.com/nixpulvis/oursh#features
- ElijahLynn 8y agoAfter watching the GIF on the README.md I was like "WHERE HAVE YOU BEEN ALL MY LIFE" (actually just the past 10'ish years)!!! Then I was like, I bet quite a few ancient wizards out there have some pretty amazing techniques they have picked up over the year. And that it would be great to be a "fly on the wall" watching some of these wizards do simple tasks. And that a YouTube channel watching them do simple tasks could be quite revealing and therefore very educational. Paul Irish did something like this a few years ago and I still use a bunch of the tools he introduced into my purview. Really, really improved productivity!
- catacombs 8y ago> And that a YouTube channel watching them do simple tasks could be quite revealing and therefore very educational. Does such a channel exist? All I've ever found has been tutorials. I want to watch unix veterans use the tools. I've found that to be the best way to learn. Hell, that's how I learned vim.
- kapad 8y agowhich channel did you use to learn vim?
- catacombs 8y agoNo channels specifically. I've watched these videos as a good primer: https://www.youtube.com/watch?v=qsWY-8n9igM https://www.youtube.com/watch?v=qsWY-8n9igM https://www.youtube.com/watch?v=_NUO4JEtkDw https://www.youtube.com/watch?v=_NUO4JEtkDw https://www.youtube.com/watch?v=blpCz9HsUaA https://www.youtube.com/watch?v=blpCz9HsUaA
- greggyb 8y agoNot a series, but a great one-off video by Gary Bernhardt, A Raw View into my Unix Hackery, linked below. I haven't bought his courses, but based on what I've seen of his talks and free material, they may get you part of the way (but I don't think all of the way) to what you're describing. If anyone has taken his courses, I'm sure they could give better feedback on those. Either way, watch the video. https://vimeo.com/11202537 https://vimeo.com/11202537
- agumonkey 8y agoMassive bravo. Thought about the need for this for ages.
- Too 8y agoNovel and interesting idea but honestly i can't see any other use for this than grep and that point you might as well just pipe the output straight into vi, less or your other favourite editor where incremental search is one keystroke away. Sure, the shell has some powerful tools like jq that editors usually lack but otherwise several levels combo piping through cut, wc, awk, etc are quite rare and are usually only done in crazy shell scripts that actually should be rewritten in more appropriate language. And the dangers of automatically running any commands shouldn't be downplayed. If you only whitelist a few programs this would be much safer, most users would only use this for grep. Or run the whole thing through a docker container.
- arendtio 8y agoSounds like fun, but I think there are other ways to achieve a similar workflow without an extra tool. For example, I like to use a tmux session with two panes. On the one side there is a `vim myscript.sh` and on the other side there is a `watch myscript.sh`. Very similar result and using rm is not that dangerous as you can finish writing your code and it gets executed as soon as you save the file (if you don't like the watch delay you can use one of the many inotify based watch tools or set -n 0.1 or something that suits your needs).
- theossuary 8y agoI use `entr` for a similar effect, little nicer than watch, though I don't think it's available on all OSes.
- ygra 8y agoOne thing that probably would be useful is when commands are not completely typed or parametrized at the end of the pipeline to still show the last result along with the error message instead of replacing everything and thus removing any context the user had.
- newnewpdro 8y agoWow, how reckless! running partial commandline constructions on every keystroke.. The obvious thing to do is only run what's been typed so far on a trigger key like tab or something.
- daenz 8y agoReally cool. The key principle that this tool highlights is the advantage of rapid iteration. Whenever you have a problem, spending the time up front to shorten your test iterations will pay off massively in the long run of the problem's life.
- msravi 8y agoFor mac: go get -u github.com/akavel/up
- pareshverma91 8y agoLooks great. Personally, I have been using a "cache" function to cache output of expensive commands. This has helped me iterate faster on pipelines that call into apis. function cache { cmd="$*" name=$(echo "$cmd" | base64) if [[ -f ./cache/$name.exit ]] && [[ $(cat ./cache/$name.exit) -eq 0 ]]; then echo "cached" else mkdir -p ./cache eval "$cmd" > ./cache/$name.out echo $? > ./cache/$name.exit fi cat ./cache/$name.out } cache ls /
- zbentley 8y agoI use something similar, created by a friend of mine, called 'bash-cache': https://bitbucket.org/dimo414/bash-cache/src/default/ https://bitbucket.org/dimo414/bash-cache/src/default/ It hooks specific functions in Bash and keys off of their arguments.
- kapad 8y agoI like this. I currently do the "plumbing" the OP refers to using files. This has generally worked for me, but want to give this tool a test run and see if it's faster than my current approach and this cache command.
- amarant 8y agothis looks great! installing! my one gripe is that the name really should have been marIO ;)
- spark888 8y agoThere is a vim plugin that enables "live preview" for python : https://github.com/metakirby5/codi.vim https://github.com/metakirby5/codi.vim
- nailer 8y ago> This is achieved by boosting any typical Linux text-processing utils such as grep, sort, cut, paste, awk, wc, perl, etc., etc., by providing a quick, interactive, scrollable preview of their results. lshw | grep network -A2 | grep : | cut -d: -f2- A better idea would be to work on tools that have structured output, so people can select the keys they want and don't have to scrape. $ Get-NetAdapter | where status -eq 'up' | select InterfaceDescription InterfaceDescription -------------------- Qualcomm Atheros QCA61x4A Wireless Network Adapter