5 ms·

It looks like it’s ostensibly a way of being able to dynamically peak into stuff that’s being passed through a pipe. A bit like if ‘pv’ (https://linux.die.net/

It looks like it’s ostensibly a way of being able to dynamically peak into stuff that’s being passed through a pipe. A bit like if ‘pv’ (https://linux.die.net/man/1/pv https://linux.die.net/man/1/pv) and a pager (eg ‘less’/‘more’/etc) had a child. I’ve typically solved this kind of problem with ‘tee’ but this looks a much cleaner approach. It could be a very useful tool for any Sysadmins who still find they need to occasionally SSH into a server for real time log parsing (which is a dying habit these days). Or for DevOps and Developers needing to debug.

cookiengineer 5y ago
Wanted to add websocketd [1] to your list. It's an amazing tool to stream debugging logs to another system where you can build your webapps that accumulate alerts. Use it only for debugging builds and not for production builds (obviously). [1] https://github.com/joewalnes/websocketd https://github.com/joewalnes/websocketd
stavros 5y ago
Ah, thank you. My main question now is whether it decides what to show based on what it's already showing (ie deduplicates somehow) or is just straight-up sampling.
- hnlmorg 5y ago
  I wasn’t entirely sure of that myself but it does look as if you can filter results manually
jewel 5y ago
Another way to simulate the behavior of this tool, if I'm understanding you correctly, is with software flow control, usually mapped as ctrl-s for XOFF and ctrl-q for XON. That lets you pause output while you look at it, and then resume once you're done.
- hnlmorg 5y ago
  You cannot add filters et al with them though. I guess this is comparable to the scroll back history on a terminal multiplexer
- kazinator 5y ago
  XOFF doesn't discard input; if you pause for too long, things will backlog, which could be a problem if we are reading directly from some program (rather than, say, reading the output of something disposable and disconnected from the source, like tail -f). When pw isn't updating the display, it's still streaming and discarding the data, which is a big semantic difference. It means that the source of the data will not block, for one thing. But if you need to see all the data or search through it, then pw is the wrong tool, at least by itself. You need to make other arrangements to capture the complete data.
dylan604 5y ago
>(which is a dying habit these days). Hmmm, I don't know what that says about me. Checking logs is 2nd nature to me. I guess that makes me old if the "kids these days" are doing something else, but what is the something else? Just not looking at the data and spinning wheels debugging? New tools available? I'll admit I've never looked for anything other than the location of a log of a service I'm investigating.
- adictator 5y ago
  We use a log aggregator and watch logs from hundreds of servers in a searchable & filterable interface like Kibana. We also use Loki and Elastic extensively. If you have money to spend, I think you'd be looking at Splunk or Datadog. As a matter of fact, we do not have SSH access to our servers. Everything happens over a web interface that has been secured.
- hnlmorg 5y ago
  Usually you'd aggregate those logs to a centralised location and have some kind of visibility platform to parse them like Splunk, ELK, DataDog, etc. That same pipeline would also typically share at least part of the process for streaming your logs to a tamper proof archive. So it's not that checking logs is a dying habit, it's process of SSHing onto the server and running shell commands to query them. Though I'm not suggesting the latter is bad practice. In some domains it's suboptimal but in other domains it is a pretty reasonable response to fault diagnosis. IT is a broad field :)
  - dylan604 5y ago
    >IT is a broad field :) Yes it is. Today's assumption seems to be that everything is being done "web scale" with hundreds of servers in multiple datacenters across the globe. I'm not so fortunate/unfortunate (depending on one's viewpoint) of having to deal with that kind of scale. I have less than 10 servers I maintain so a lot of these new fancy viewers is overkill for me.
    - fragmede 5y ago
jwilk 5y ago
PSA: don't use linux.die.net. It's horribly outdated. This particular man page is from 2012 or earlier.
- hnlmorg 5y ago
  That doesn't really matter in this instance, I just wanted some example of what pv was as it's a lesser known utility and that page happened to be the top non-spammy link in DDG.
kazinator 5y ago
> for real time log parsing Or other things! tcdump -i <if> -l | pw # -l for line buffering Now you have an interactive network monitor that, in some ways, has better ergonomics than Wireshark. Want IPV6 only? :g IP6 done. Okay, only packets from bob to alice: :g bob.*>.*alice Only UDP: :g UDP Forget UDP, how about http: :r :g http Kind of thing.

cookiengineer 5y ago

Wanted to add websocketd [1] to your list. It's an amazing tool to stream debugging logs to another system where you can build your webapps that accumulate alerts. Use it only for debugging builds and not for production builds (obviously). [1] https://github.com/joewalnes/websocketd https://github.com/joewalnes/websocketd

stavros 5y ago

Ah, thank you. My main question now is whether it decides what to show based on what it's already showing (ie deduplicates somehow) or is just straight-up sampling.

hnlmorg 5y ago
I wasn’t entirely sure of that myself but it does look as if you can filter results manually

jewel 5y ago

Another way to simulate the behavior of this tool, if I'm understanding you correctly, is with software flow control, usually mapped as ctrl-s for XOFF and ctrl-q for XON. That lets you pause output while you look at it, and then resume once you're done.

hnlmorg 5y ago
You cannot add filters et al with them though. I guess this is comparable to the scroll back history on a terminal multiplexer
kazinator 5y ago
XOFF doesn't discard input; if you pause for too long, things will backlog, which could be a problem if we are reading directly from some program (rather than, say, reading the output of something disposable and disconnected from the source, like tail -f). When pw isn't updating the display, it's still streaming and discarding the data, which is a big semantic difference. It means that the source of the data will not block, for one thing. But if you need to see all the data or search through it, then pw is the wrong tool, at least by itself. You need to make other arrangements to capture the complete data.

dylan604 5y ago

>(which is a dying habit these days). Hmmm, I don't know what that says about me. Checking logs is 2nd nature to me. I guess that makes me old if the "kids these days" are doing something else, but what is the something else? Just not looking at the data and spinning wheels debugging? New tools available? I'll admit I've never looked for anything other than the location of a log of a service I'm investigating.

adictator 5y ago
We use a log aggregator and watch logs from hundreds of servers in a searchable & filterable interface like Kibana. We also use Loki and Elastic extensively. If you have money to spend, I think you'd be looking at Splunk or Datadog. As a matter of fact, we do not have SSH access to our servers. Everything happens over a web interface that has been secured.
hnlmorg 5y ago
Usually you'd aggregate those logs to a centralised location and have some kind of visibility platform to parse them like Splunk, ELK, DataDog, etc. That same pipeline would also typically share at least part of the process for streaming your logs to a tamper proof archive. So it's not that checking logs is a dying habit, it's process of SSHing onto the server and running shell commands to query them. Though I'm not suggesting the latter is bad practice. In some domains it's suboptimal but in other domains it is a pretty reasonable response to fault diagnosis. IT is a broad field :)
- dylan604 5y ago
  >IT is a broad field :) Yes it is. Today's assumption seems to be that everything is being done "web scale" with hundreds of servers in multiple datacenters across the globe. I'm not so fortunate/unfortunate (depending on one's viewpoint) of having to deal with that kind of scale. I have less than 10 servers I maintain so a lot of these new fancy viewers is overkill for me.
  - fragmede 5y ago

jwilk 5y ago

PSA: don't use linux.die.net. It's horribly outdated. This particular man page is from 2012 or earlier.

hnlmorg 5y ago
That doesn't really matter in this instance, I just wanted some example of what pv was as it's a lesser known utility and that page happened to be the top non-spammy link in DDG.

kazinator 5y ago

> for real time log parsing Or other things! tcdump -i <if> -l | pw # -l for line buffering Now you have an interactive network monitor that, in some ways, has better ergonomics than Wireshark. Want IPV6 only? :g IP6 done. Okay, only packets from bob to alice: :g bob.*>.*alice Only UDP: :g UDP Forget UDP, how about http: :r :g http Kind of thing.