11 ms·
UP: A tool for writing Linux pipes with instant live preview
- crvdgc 6y ago> WARNING: Please be careful when using it! It could be dangerous. In particular, writing "rm" or "dd" into it could be like running around with a chainsaw. But you'd be careful writing "rm" anywhere in Linux anyway, no? Side effects always seems like potential disasters for me.
- jackhalford 6y agoif linux had openbsd's pledge(2) then you could disable filesystem writes, which you don't need for grepping. I know linux probably has orher mechanisms to achieve the same result but in this situation pledge seems natural.
- dan-robertson 6y agoProbably that’s right. But note that sort, for example, may need to write to (temporary) files. So disabling writes entirely is probably not a sufficient solution
- 10000truths 6y agoYou can achieve the same thing in Linux with seccomp.
- onli 6y agoWhy side effects? It's about the main effect of tools like rm or dd. Some things just don't fit to live previews like provided by UP.
- nabakin 6y agoYou don't want to accidentally delete or write to things and this command seems like an accident waiting to happen. A command whitelist would be very helpful but it doesn't look like UP has one.
- dmingod666 6y agoYes, even a blacklist with obvious ones listed would be cool. Imagine just typing the characters rm and you're gone..
- michaelmior 6y agoGood point. That's a big problem. I could conceivably have a perfectly harmless tool where a prefix of the name is something dangerous.
- akavel 6y agoOne of the reasons for not doing this is that I decided I don't want to lure the users into a false sense of security. A comment by hnlmorg basically sums this up better than I could: https://news.ycombinator.com/item?id=26645875 https://news.ycombinator.com/item?id=26645875 (The other being, that I was cutting corners on my initial ideas brutally trying to push the tool out of the door, and it's still kind of a miracle I eventually managed to do that.)
- mimimi31 6y agoI remember reading about a project that let you "preview" the effects (e.g. file changes) of terminal commands by using ptrace to intercept the syscalls and just logging them instead. Maybe you could do something similar here to prevent accidental file deletions.
- nabakin 6y agoMy first thought was to sandbox the executable but this sounds even better. Is it hard to implement the interception of all syscalls? And can file reads be distinguished from file writes?
- simcop2387 6y agoNot terribly hard to do, and yes you can distinguish reads, writes, etc. A tool to log syscalls to see what might be possible is strace, it's incredibly useful for debugging a misbehaving application that either you don't have all the source for or it's gigantic codebase that you don't understand. Seeing which files a program is looking for is often all you need to figure out why it's behaving the way it is (i.e. locating an undocumented/badly documented/incorrectly documented config file)
- alvarlagerlof 6y agoWow that would be cool
- minxomat 6y agoThat's maybe: https://github.com/p-e-w/maybe https://github.com/p-e-w/maybe
- adictator 6y agoWow. That sounds like a really handy tool. Can it be used to debug programs, maybe?
- akavel 6y agoPlease note that up respects $SHELL, so if you find a way to set $SHELL to point to maybe (possibly wrapped in some helper script) in a useful manner, it might actually work! I'd highly appreciate an issue with a report/guide for that then on up, others might find that helpful! For details, see: https://github.com/akavel/up/releases/tag/v0.3 https://github.com/akavel/up/releases/tag/v0.3
- nonbirithm 6y agoChrome's developer tools have live preview, and they apparently accomplish this at the VM level by blacklisting a set of impure bytecodes when running the code in a separate evaluation context. If one of the bytecodes is found then it throws an exception. I kind of wish LuaJIT had something like that.
- haolez 6y agoIs there a way to start a new instance of bash with read-only permissions only? This could easily fix the side-effects issue.
- alexf95 6y agoI didn't realise I needed this until I saw this post. Now I'm wondering why this isn't the standard already for every command line out there. Seems like a no brainer improvment with non to few downsides.
- hnlmorg 6y agoI've been working on a way to implement this into my own shell (murex). The shell already has a builtin support for known "safe" commands, that is commands that are generally safe for background execution. The reason murex has this feature is because it supports executing the pipeline to generate more relevant autocompletion suggestions (eg when picking out nodes from a JSON or YAML file). Thus implementing an `up` like feature should be a pretty easy addition....however I ran into a bug that stalled the development of said feature and it's been in that stalled state for the last ~2 years. The reason development stalled for so long is because personally I'm not sure this is something that should be widely supported. Even with murex, which has an internal catalogue of known safe(ish) commands, the dangers of tools like this is pretty high. Running this against a regular shell which doesn't even do that seems like potential suicide.
- rocqua 6y agoHow about spinning up a container with the entire host file-system mounted in a Copy-on-Write configuration. This way you can trash the container as much as you want without affecting anything on the outside. Combine it with syscall logging and you might actually have a great tool for figuring out exactly how a command is affecting your system.
- hnlmorg 6y agoThat'll certainly help but you'd still have issues with any connected devices (USB, network API alls, etc). I don't think it is a solvable problem though. There is no way to know ahead of time which usage is safe and which is destructive because of the variety of UNIX-like configurations out there, variety of coreutils and variety of optional flags. Not to mention all the 3rd party tools out there. And some seemingly destructive usages are vital for safe usage (eg tmp files). Worse still, by the time you build a safe container you've have spent more time spinning up a sandbox than you've had spent using the CLI preview tools anyways (yeah ZFS could help here but few people run ZFS on their dev machines and you still haven't solved mocking APIs for connected devices). I just can't see how this can be achieved safely without completely redesigning how SYSCALLs work from the ground up to follow a method more akin to iOS or FireFox's on-demand permissions. At which point you've gone well beyond the realm of "yak shaving"
- dan-robertson 6y agoThis looks quite good, and I’ve wanted something like this for a while now. I wonder if it could be improved by incremental versions of commands like grep, e.g. making the change grep net -> grep network faster. Another system with some different but related capabilities is userland: https://hisham.hm/userland/ https://hisham.hm/userland/
- rakoo 6y agoThat is some impressive Proof of Concept and it's not even surprising considering who created it. Using a "terminal" like that would probably change a lot of how we work on the command-line
- thitcanh 6y agoWow this is almost exactly what I’ve been looking for for years. I don’t want a tool like up that works on a fixed entry command; I want a full terminal that takes care of everything, showing me the output at every step. What’s missing is some caching (think HTTP requests) and limitation (e.g. evaluate the first 4 HTTP calls, and wait for me to approve the rest if they’re good)
- akavel 6y agoA variation on userland is the next project I'm trying to work on, though there are... distractions. In a random case that some benevolent millionaire wants to sponsor me to work full time on things like that, I'm certainly open to being contacted, my email is in the commits ;) As to another huge and absolutely awesome emerging contender in this space, see https://luna-lang.org https://luna-lang.org
- ducktective 6y agoNow, that one was a futuristic tool! Thanks for sharing! (He also wrote htop :D) https://github.com/hishamhm https://github.com/hishamhm
- dakra 6y agoI use a similar tool in Emacs called piper https://gitlab.com/howardabrams/emacs-piper https://gitlab.com/howardabrams/emacs-piper And here's the EmacsConf video for those interested: https://www.youtube.com/watch?v=HKJMDJ4i-XI https://www.youtube.com/watch?v=HKJMDJ4i-XI
- marco_craveiro 6y agoPiper looks amazing, shame its not on a package repo.
- akavel 6y agoI didn't manage to grasp it from quickly skimming the video and the gifs: is it possible to retry a command in piper until I find the right combination of flags, and only then press `|` to push it to the buffer? Ideally, keeping the incomplete command visible for editing all the time? Also, can I get back to older contents of the buffer, i.e. a couple `|`s back? If yes, that might be basically what I wanted up to be from the beginning.
- ducktective 6y agoThe developer, Mateusz, did a "show HN" some 2.5 years ago: https://news.ycombinator.com/item?id=18291302 https://news.ycombinator.com/item?id=18291302
- akavel 6y agoHi ducktective, thanks for submitting again! :) It was a pleasant surprise seeing it here today, on my regular HN addiction^H^H^H daily read :) I completely didn't connect the dots that it's ok to repost periodically to share the fun and serve as a refresher for the lucky 10k, kudos to you for being here to do that :)
- ducktective 6y agoI kinda hunt for cool UNIX tools periodically :D Someone mentioned it in a recent post on fzf, I checked out the repo and posted here. Took me a while to get the lucky 10k reference ;) https://xkcd.com/1053/ https://xkcd.com/1053/
- akavel 6y ago'twas me ;P and the timing coincidence was suspicious indeed xD
- alexsuslin 6y agoHN should have a separate section and do the annual re-post of the most cool stuff =)
- kreetx 6y agoThere was this same idea realized via fzf preview, also posted to hn I think.
- WalterGR 6y agofzf has been submitted many times - https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=fzf&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... - but most recently: Improving Shell Workflows with Fzf (jambor.dev) https://news.ycombinator.com/item?id=26634419 https://news.ycombinator.com/item?id=26634419 221 points by todsacerdoti 22 hours ago | 51 comments
- alpaca128 6y agoJust recently I thought I could use a live checking tool for regexes but without the horrible UI of the typical website made for this purpose. I will definitely try this and see how it combines with ripgrep/rg.
- dmingod666 6y agoregex101.com is pretty comprehensive, my go to for complex regexes.. not sure if that's what you're complaining about..
- alpaca128 6y agoThat's what I tried recently among others, it's the website equivalent of an IDE. And when I actually started entering some regex it started throwing errors and timeout messages for inputs that standard grep will happily chew through in fractions of a second. On subpar hardware that site is a single collection of annoyances, including window-filling loading screens. That said I tried UP and it works very nicely for what I need.
- foreigner 6y agoWow I literally just thought about building a tool like this yesterday! Spooky.
- rgrau 6y agoFor fast iteration with pipes, using only basic unix tools, and arguably less risky to fat-fingering, I write it in a file, and use watch -n1. Easy to modify, temporarily comment parts of it, etc... Also, I wrote this plog function (for pipe-log, or something): plog() { local msg=${1:-plog} tee >(sed -e "s/^/[$msg] /" | cat 1>&2 ) } That allows me to see what's happening in the middle steps of the pipeline. I ended up compiling a list of tricks like that in https://raimonster.com/scripting-field-guide/ https://raimonster.com/scripting-field-guide/ , in case anyone is interested (and feedback is appreciated)
- maxwelljoslyn 6y agorgrau, I'm halfway through your field guide. I appreciate how concise the text portions are, and how, although each section teaches one thing at a time, you are sometimes willing to leave the reader with "here are all the major incantations, figure out the details of each one and appreciate them" - such as in section 4.4. Learned several new bashisms which i will put into my computer tip memorization deck. Thank you for writing.
- jolmg 6y agoYou can skip cat. tee >(sed -e "s/^/[$msg] /" 1>&2)
- rustyminnow 6y agoThis is super cool. There's so much good stuff in there I'm gonna have to come back a few times just to digest it all! Also looking forward to some of the links at the bottom. Do you have a standard template you use for your scripts? You kind of touch on this in section 4.20.1 but don't go too deep. Also have you done any work with writing zsh (or bash) completions? I feel like having good completions would make my scripts magnitudes more handy, but I've had trouble finding good starter resources.
- whizzter 6y agoFeels like a perfect tool for taco-bell style programming(googleable), and i think there needs to be more in this style. Powershell for example has an IDE(MS style) that is damn helpful by providing breakpoints/intraspection but this sits at a lower/faster level for just inspecting the initial pipings.
- pjmlp 6y agoPowershell is probably the closest shell to the REPL experience of Xerox PARC and ETHZ workstations. UNIX shells could approximate it with support for shared objects, D-Bus/XPC/IPC, but few make use of it.
- ducktective 6y agoI'm interested to know how and in what ways the REPL experience in Xerox PARC and ETHZ workstations were enjoyable...Can you elaborate please or share some links? I know that there is Nushell (1) that tries to re-do the concept of shell with ideas taken from Powershell. [1]: https://github.com/nushell/nushell https://github.com/nushell/nushell
- rippeltippel 6y agoI started a very similar side project (in Rust) at the beginning of lockdown, but didn't manage to keep up as I had to spend most of my "free" time parenting. Although the self-proud part of me feels a bit disappointed that someone else got there first, I'm actually glad that it happened (it also means that my idea was sound). I didn't even realise that something like that existed already. Kudos to both the developer and the OP!
- dmingod666 6y agoJust saying, the tool is about 3 years old, most recent commit is 12 months ago..
- rippeltippel 6y agoI noticed, that's why I wrote "I didn't even realise that something like that existed already" :) And I'm grateful to the OP who brought this tool to HN's attention.
- michaelmior 6y agoActually, 5 months ago :)
- akavel 6y agoDon't worry, when I published up I also learned about a previous similar project from commenters ;) probably wouldn't start if I knew earlier ;) fortunately I didn't know :D just in case if you still have some mojo left, up is very open to an idea of a RIIR contender, it's actually even kinda encouraged in the readme :)
- mtbkvc 6y agovery good idea. thanks akavel
- qrbLPHiKpiux 6y ago“Running around with a chainsaw“ Lol!
- zachlatta 6y agoThis seems amazing. I can’t wait to try this.
- orf 6y agoSlightly related, I wrote a tool called ptail[1] that only shows the last N lines of output from a pipe. Quite useful in some situations 1. https://github.com/orf/ptail https://github.com/orf/ptail
- deleted 6y ago[deleted]
- Saint_Genet 6y agoMaybe I'm just old and grumpy, but what do you gain by this compared to just building up your pipe directly in the shell? Not having to hit enter once in a while?
- jolmg 6y agoYou still have to press Enter. I'm with you. I can't see myself preferring this over using my shell's interactive prompt, history, and a use of head/less at the end. Thanks to the author and OP for sharing and it's good for other people if they find this useful, though. EDIT: Maybe I could find this useful when the command that provides `up`'s input is not something that can be repeated for the same input. But even then, you can put it in the background with its output redirected to a file, then make the command that you would build inside `up` from a pipe like `tail -fn+1 bg-job.out | cmd`.
- akavel 6y agoNot much, really. Yet somehow those little things add up to something... faster :) Basically "just": - not having to append `| head -n $ROWS` or something similar on every command (yet having the view as-if-you-typed-it, but at the same time being able to scroll down further through the results; and having them visible as you continue tweaking, unlike `less` which you have to quit to tweak the pipeline), - not having to press up-arrow-key, then alt-b / alt-f to a position in the pipeline you want to tweak after every Enter (e.g. just before the `| head...`, but sometimes a bit more back), - sometimes also not having to dump the initial input from a command into a file, and to wait until it's finished, but being able to immediately start filtering even while the input is still being generated. Once I just had those ideas, I felt pain every time I repeated the dance, enough that it made me push on up even when I was super tired of working on it. The pain of not having it when I know it could exist was even bigger.
- uberduper 6y agoWhen whittling down text output like the examples on the project page, I've found that just piping into vim and using a combination of vim commands and external commands like `:%!grep -v foo` to be far easier and much more capable.
- mraza007 6y agoI was looking for a very similar tool for my project. Anyhow does some one know about a tool that allows you to render your commandline output to html
- ducktective 6y agoDepends on what markup language that output is in. If you just want a conversion of colors and ANSI escape codes to HTML tags, I've recently found and used this script (1). This SO link discusses few other options (2). [1]: https://github.com/pixelb/scripts/blob/master/scripts/ansi2html.sh https://github.com/pixelb/scripts/blob/master/scripts/ansi2h... [2]: https://unix.stackexchange.com/questions/44956/converting-colored-output-into-html https://unix.stackexchange.com/questions/44956/converting-co...
- mraza007 6y agoThanks for sharing these and I think these really help
- thesh4d0w 6y agoI thought I'd seen almost everything in bash, but I've never seen |& before. My attempts at googling it aren't coming up with anything, can anyone advise what it does compared to a normal pipe?
- amarshall 6y agoFrom https://www.gnu.org/software/bash/manual/html_node/Pipelines.html https://www.gnu.org/software/bash/manual/html_node/Pipelines...: > If ‘|&’ is used, command1’s standard error, in addition to its standard output, is connected to command2’s standard input through the pipe; it is shorthand for 2>&1 |. This implicit redirection of the standard error to the standard output is performed after any redirections specified by the command.
- thesh4d0w 6y agoAh ha, I suspected it was that but couldn't find the relevant docs. Thanks!
- amarshall 6y agoSomewhat similar is fzz https://github.com/mrnugget/fzz https://github.com/mrnugget/fzz
- Siira 6y agoI find `fzf --bind "change:reload:some-shell-command {}" --disabled` a better alternative.