6 ms·
Show HN: How did I live without Pipe Watch?
- therein 5y agoSo it is like the edge trigger functionality of a Digital Storage Oscilloscope for pipes? I can think of more than a handful of times this would have become handy but also seems like a very unlikely tool that I find myself reaching, especially due to the complicated usage.
- marto1 5y ago> especially due to the complicated usage. Said he while using find, dd, rsync, tar and others every day. But in all seriousness new cmd tools should be given a chance, it sometimes feels like people believe posix tools were perfected in the 80s and further development makes no sense.
- kazinator 5y agoBut in actual fact, it is very easy to use. You might not get that from the video because it's inspired by Vi: commands are not echoed, other than colon commands and search patterns. So in the video you don't always know what is going on: did the display change because of a command you don't see or was it spontaneous? That is all painfully obvious if you're in the driver's seat.
- vosper 5y agoI don’t want to watch the video on my phone, so would you please give some practical examples of what this might be used for? I read the description and I don’t know what it’s for.
- caddywompus 5y agoI've only watched the video, so I don't know a lot about the tool. But from what I can tell, it's a really neat way to view text as it's streaming through a pipe, _and_ it keeps a line at the bottom, for you to be able to type a line, which when you press return, sends your text to the STDIN of the program that is streaming the text. This can relieve a pain point if you ever need to work on a serial port, and as you are typing in a command, the other side sends text, which overwrites the characters you were typing (depending on how the local echo is configured). It reminds me of a line based serial terminal emulator, where the input is a separate text box, and the output is a larger box above it. Oh, and it also seems to have methods of sampling the data as it comes in, so that it can be configured to only show the messages you are looking for, and also counting them? The source file is also tiny: https://www.kylheku.com/cgit/pw/tree/pw.c https://www.kylheku.com/cgit/pw/tree/pw.c Very Unix
- stavros 5y agoI don't like the landing page, it lists all the niceties of the program and details how it does what it does, but it doesn't say what it does. Why do I need this? What problem does it solve for me? I know it samples pipes, I don't know why I would want something that does.
- hnlmorg 5y agoIt looks like it’s ostensibly a way of being able to dynamically peak into stuff that’s being passed through a pipe. A bit like if ‘pv’ (https://linux.die.net/man/1/pv https://linux.die.net/man/1/pv) and a pager (eg ‘less’/‘more’/etc) had a child. I’ve typically solved this kind of problem with ‘tee’ but this looks a much cleaner approach. It could be a very useful tool for any Sysadmins who still find they need to occasionally SSH into a server for real time log parsing (which is a dying habit these days). Or for DevOps and Developers needing to debug.
- cookiengineer 5y agoWanted to add websocketd [1] to your list. It's an amazing tool to stream debugging logs to another system where you can build your webapps that accumulate alerts. Use it only for debugging builds and not for production builds (obviously). [1] https://github.com/joewalnes/websocketd https://github.com/joewalnes/websocketd
- stavros 5y agoAh, thank you. My main question now is whether it decides what to show based on what it's already showing (ie deduplicates somehow) or is just straight-up sampling.
- hnlmorg 5y agoI wasn’t entirely sure of that myself but it does look as if you can filter results manually
- jewel 5y agoAnother way to simulate the behavior of this tool, if I'm understanding you correctly, is with software flow control, usually mapped as ctrl-s for XOFF and ctrl-q for XON. That lets you pause output while you look at it, and then resume once you're done.
- anned20 5y agoDescription from man page: pw stands for Pipe Watch. This is a utility which continuously reads textual input from a pipe or pipe-like source, and maintains a dynamic display of the most recently read N lines. If pw is invoked such that its standard input is a TTY, it simply reads lines and prints them in its characteristic way, with control characters replaced by caret codes, until end-of-file is encountered. Long lines aren't clipped, and there is no interactive mode. The intended use of pw is that its standard input is a pipe, such as the output of another command, or a pipe-like device such as a socket or whatever. In this situation, pw ex- pects to be executed in a TTY session in which a /dev/tty device can be opened, for the purposes of obtaining interactive input. The remaining description pertains to this in- teractive mode. In interactive mode, pw simultaneously monitors its standard input for the arrival of new data, as well as the TTY for interactive commands. Lines from standard input are placed into a FIFO buffer. While the FIFO buffer is not yet full, lines are displayed immedi- ately. After the FIFO buffer fills up with the specified number of lines (controlled by the -n option) then pw transitions into a mode in which, old lines are bumped from the tail of the FIFO as new ones are added to the head, and refresh operations are required in order to display the current FIFO contents. The display only refreshes with the latest FIFO data when 1. there is some keyboard activity from the terminal; or 2. when the interval period has expired without new input having been seen; or else 3. whenever the long period elapses. In other words, while the pipe is spewing, and there is no keyboard input, the display is updated infrequently, only according to the long interval. The display is also updated when standard input indicates end-of-data. In this situation, pw terminates, unless the -d (do not quit) option has been specified, in which case it pw stays in interactive mode. The the end-of-data status is indicated by the string EOF being displayed in the status line after the data.
- chris_wot 5y agoHow is this different to tail -f ?
- 5y ago
- chris_wot 5y agoIs this like pipe view? This was used as an example on monitoring a MySQL restore. The command was: pv sqlfile.sql | mysql -uxxx -pxxxx dbname It seems that pv copies the file to standard output, MySQL then consumes this. The normal command is mysql -u user -p pass < sqlfile.sql But this shows no progress output.
- mbreese 5y agoNo. pv shows a progress bar when data is moving through a pipe. This samples data as it moves through a pipe. The difference being, pv is more like a fancy version of cat (and more, but that’s the primary use case). This samples data that is being piped around, so you can see the actual data, not just the progress.
- miked85 5y agoIt's pretty difficult to figure out what this even does or what the benefit is.
- bjarneh 5y agoThis is super handy. I've often found myself 'tail'-ing logs to 'grep' to filter out stuff; pw makes that process 10 times better. Very cool.
- natrys 5y agoSeems like similar to lnav[1] or angle-grinder[2]. [1] https://lnav.org/ https://lnav.org/ [2] https://github.com/rcoh/angle-grinder https://github.com/rcoh/angle-grinder
- yewenjie 5y agoCan it do the following? I have a dev server which is spitting lots of logs and I want to interactively search the logs with something like `fzf`. No changes can be made to the dev server.
- kazinator 5y agoBut suppose you're debugging some scenario (e.g. particular kind of request that generates some logs according to its access/interaction pattern). You can drill into this: interactively determine what things to filter for, trigger the display snapshot/update on particular multi-line patterns and such. You can have it so that whenever you trigger the use case, the new instance of the log appears in the display and freezes, and contains just the relevant information.
- anotherhue 5y agoTail the logs into fzf --no-sort --tac
- gh123man 5y agoI keep this alias in my zshrc for exactly this purpose alias lzf="fzf +s --tac --bind 'enter:select-all+accept' -m" With the added bonus that you can press return to release the filtered lines to stdout. Really useful for interactively filtering logs.
- kazinator 5y agoNo. pw works strictly with a small window of live data. It doesn't assume that the file descriptor being read is a seekable file. It doesn't hoard all the data for searching. If there is some dynamic pattern in the logs that repeats (or that you can cause to repeat by executing some test steps on the software which produces the logs), you may be able to use pw to latch on to that pattern. E.g. suppose there is detailed function enter/leave tracing in the log. You can discover that certain functions are traced when you do something, and then filter for that material to remove unrelated spewage, and perhaps set up a trigger to capture the scenario you are reproducing. Once you know that, you can use other tools to find the same thing. You know what to look for in off-line analysis of a complete log. This is not just for logs. If you pipe tcpdump -i <eth> -l | pw you basically get an interactive network monitoring utility. E.g. add: :g thishost.*>.*thathost and now you're just seeing packets from thishost to thathost.
- iib 5y agoThis seems like a less-interactive `lnav`, somewhere between `lnav` and `tail -f`, which is great for me to see, as I think this problem-space may be under-researched.
- jewel 5y agoI'd love to have a tool to pipe into that removes low-entropy content automatically, perhaps with machine learning. For example, last week I was grepping through some security logs looking for anomalous logins. There were a bunch of other security events in the log file. Eventually my command looked like `cat /var/log/auth.log | grep -v Disconnected | grep -v cron:session | grep -v 'Invalid user'` and so forth until I only had the "interesting" lines left. Maybe the right approach to this tool would be to put the lines into a tree based on the longest substrings.
- xorcist 5y agogrep takes multiple arguments so if you want fewer processes then: grep -v -e Disconnected -e cron:session -e "invalid user" /var/log/auth.log would also suffice (or you can use -E which would enable the usual "Disconnected|invalid|cron").
- pabs3 5y agoAfter watching the demo video for PW, I think it is better than what you were doing, but doesn't have the automation you want.
- kazinator 5y agoIt may be a good way to discover what patterns are worth looking for, and then encoding that into a grep. You need grep if you want to keep everything that matches; no skipping. But the cycle of repeatedly killing "tail -f | grep ..." and then trying different patterns is a little less ergonomic than doing it inside one program. If the input is coming from a pipe directly, trying a different set of grep patterns means killing and restarting the program.
- upwardbound 5y agoIf I understand what it's doing correctly, this reminds me of the trigger function of an oscilloscope, which "enables repetitive waveforms to be displayed on the screen in steady fashion." https://www.electronics-notes.com/articles/test-methods/oscilloscope/oscilloscope-trigger.php https://www.electronics-notes.com/articles/test-methods/osci...
- kazinator 5y agoYes. I thought about changing the name to Pipe Scope after the trigger feature was added, but then people would bug me to make it dual trace, and extend the bandwidth to 100 MHz or more and such.
- vishnugupta 5y agoThis looks same as the older "watch" command that I use once in a while. From the man page. "watch - execute a program periodically, showing output fullscreen". I use it as "watch -5 -d <command>". "-5" is to run the "command" every 5 seconds and "-d" is to highlight the difference from the previous output.
- kazinator 5y agoThe "Watch" part of the name is inspired by the watch command. Something is being periodically redisplayed, namely the contents of a FIFO buffer of lines of text.
- kazinator 5y agoNext piece of development; - pw currently misbehaves if you Ctrl-Z suspend it and then try "bg". It doesn't handle the SIGTTOU signal telling it that it's not allowed to send to the TTY due to being in the background. (It won't die, but you may have to suspend it again and do "fg"). - the ideal behavior would be: if we Ctrl-Z, bg, the program runs and just reads the pipe in the background. With that functionality, we can take any program that spews logs onto standard output and just throw it into the background indefinitely without having it block. $ spewing-program | pw & [1] 123 $ bg [1]+ spewing-program | pw & To see the latest bit of logging from the program, we foreground it. A program which produces detailed logging that cannot be turned off or adjusted dynamically will benefit from this.
- dundarious 5y agoI would recommend adding a license as well.
- kazinator 5y agoThanks for pointing that out! I could have sworn I had put a BSD copyright block at the top, but now I see it's not there. I did that to a different program and misremembered. No, scratch that; I did that in the man page source code, but forgot the C source file. [Issue fixed now.]
- deltaonefour 5y agoI'm tired of unix interfaces. Really. It's great to learn a few like vim and grep. But every single tool requiring you to remember a bunch of commands is just too much. Great this tool looks cool. The UI turns me off. Just like almost every Unix UI. It's not about how good the UI is either. It's about the fact that every new tool requires reading MAN pages. More tools need to be like nano because I simply don't have time to read the MAN for every single thing.
- kazinator 5y agoBy the way, in the video nothing is initially happening after some traces are shown. Why is that? It's not that the strace isn't producing any new output, but rather because strace is spewing rapidly, and the default timeout is one second. Therefore, the pipe never times out, which means that the time-out-driven refresh isn't seen. Only the 10 second long interval refreshes the display automatically. Before setting setting the trigger, I unpaused the display. Every input event from the TTY also refreshes the display in free-running mode. So as I'm typing /poll[Enter], there is display activity with each key. Then when the trigger is set, things start going due to the frequent triggering. Things might have been less confusing at the start of the video if I had shortened the long timeout.
- moonbug 5y agothat is a terrible page. it leads with a while bunch of stuff no one cares about, and then goes on ti completely fail to convey what the tool actually goes.