5 ms·

Does anyone know an easy way to tail follow two logs at once on a pair of load balancing servers?

by hirundo 5y ago

deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
jolmg 5y ago
mkdir foo bar sshfs foo.server:foo.dir/ foo/ sshfs bar.server:bar.dir/ bar/ tail -F foo/foo.log bar/bar.log A method that doesn't involve sshfs: servers=(foo bar) pipes=("${servers[@]/%/.pipe}") mkfifo "${pipes[@]}" tail -F "${pipes[@]}" & for p in "${pipes[@]}"; do true > "$p" done for s in "${servers[@]}"; do ssh "$s" tail -F file.log > "${s/%/.pipe}" & done %tail There's timing issues on this alternative. I just ran each statement separately in a prompt, and luckily it worked. The first `true` statement needs to run after `tail` has started trying to read the first argument, and once that dies, the second run of `true` needs to run after `tail` starts reading the second argument. Last time I did something like this, I just typed the statements without the loops and ran by hand.
- dmurray 5y ago
  Wow, I never used sshfs but this seems like a fantastic convenience.
gorgoiler 5y ago
You can ssh host1 tail -F log | sed -e s/^/host1:/ But you might eventually want to log everything to a third host. Use syslog to send logs across the network and configure your local software to log to syslog. When you have 10k hosts then sampling 1 in N log lines is still useful as a canary, while avoiding being a firehouse for the central log sampler. If something heinous occurs you can get the unsampled log from the original machine, if you need it. Much more sophisticated techniques abound, and others will comment I’m sure, but it’s also nice to have the syslog trail as a fallback. You never know in advance what’s going to fail and syslog will show you everything from your Apache logs to kernel bugs and faulty NICs.
- jolmg 5y ago
  > You can ssh host1 tail -F log | sed -e s/^/host1:/ That's only for one host. If you mean something akin to: tail -F <(ssh ...) <(ssh ...) that wouldn't work, because `tail` would wait for the first `ssh/sed` to close their output (to reach the end of the first pipe) before starting to read the second `ssh/sed`'s output.
  - a1369209993 5y ago
    You can do: $ ssh ... >a.log & $ ssh ... >b.log & $ tail -F a.log b.log
    - jolmg 5y ago
      You're right. That's pretty practical. I guess I try too hard to avoid writing to disk when I can.
retzkek 5y ago
Honestly Loki and Promtail are so easy to setup, and jump into not just tailing and searching but incorporating logs into Grafana dashboards (you are using Grafana already to monitor host and service metrics, right?), that they make sense to run for everything but the smallest scale deployment (that is, a single service on a single node).
tstack 5y ago
Sorry to plug lnav again, but if the data volume is not "too much"[1], the latest version of lnav has support for tailing files on remote hosts that are accessible via ssh: https://lnav.org/2021/05/03/tailing-remote-files.html https://lnav.org/2021/05/03/tailing-remote-files.html So, to tail "/path/to/log" on host1 and host2, you'd start lnav with: $ lnav host1:/path/to/log host2:/path/to/log [1] - This is the initial release of this feature, so it probably has some performance issues. I'd be interested to know if it worked or not.
- fleur-de-lotus 5y ago
  I tried but my timestamp is of the form %Y%m%d.%H%M%S: Do you have an option to just specify the timestamp format? If not, can you give an example of defining own format? Many thanks
  - tstack 5y ago
    I'll add this timestamp to the set of defaults for the final v0.10.0 version later today. The documentation for the format files is here: https://docs.lnav.org/en/latest/formats.html https://docs.lnav.org/en/latest/formats.html The "timestamp-format" property in the format definition is an array of the formats to be tried beyond the builtin set. Here are some example formats written by a user: https://github.com/aspiers/lnav-formats https://github.com/aspiers/lnav-formats I think some of them define their own timestamp format.
deleted 5y ago
[deleted]
jedberg 5y ago
Use syslog to send all the logs to one place. Either a third server or just have one send to the other if you can handle the load.

Does anyone know an easy way to tail follow two logs at once on a pair of load balancing servers?

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

jolmg 5y ago

mkdir foo bar sshfs foo.server:foo.dir/ foo/ sshfs bar.server:bar.dir/ bar/ tail -F foo/foo.log bar/bar.log A method that doesn't involve sshfs: servers=(foo bar) pipes=("${servers[@]/%/.pipe}") mkfifo "${pipes[@]}" tail -F "${pipes[@]}" & for p in "${pipes[@]}"; do true > "$p" done for s in "${servers[@]}"; do ssh "$s" tail -F file.log > "${s/%/.pipe}" & done %tail There's timing issues on this alternative. I just ran each statement separately in a prompt, and luckily it worked. The first `true` statement needs to run after `tail` has started trying to read the first argument, and once that dies, the second run of `true` needs to run after `tail` starts reading the second argument. Last time I did something like this, I just typed the statements without the loops and ran by hand.

dmurray 5y ago
Wow, I never used sshfs but this seems like a fantastic convenience.

gorgoiler 5y ago

You can ssh host1 tail -F log | sed -e s/^/host1:/ But you might eventually want to log everything to a third host. Use syslog to send logs across the network and configure your local software to log to syslog. When you have 10k hosts then sampling 1 in N log lines is still useful as a canary, while avoiding being a firehouse for the central log sampler. If something heinous occurs you can get the unsampled log from the original machine, if you need it. Much more sophisticated techniques abound, and others will comment I’m sure, but it’s also nice to have the syslog trail as a fallback. You never know in advance what’s going to fail and syslog will show you everything from your Apache logs to kernel bugs and faulty NICs.

jolmg 5y ago
> You can ssh host1 tail -F log | sed -e s/^/host1:/ That's only for one host. If you mean something akin to: tail -F <(ssh ...) <(ssh ...) that wouldn't work, because `tail` would wait for the first `ssh/sed` to close their output (to reach the end of the first pipe) before starting to read the second `ssh/sed`'s output.
- a1369209993 5y ago
  You can do: $ ssh ... >a.log & $ ssh ... >b.log & $ tail -F a.log b.log
  - jolmg 5y ago
    You're right. That's pretty practical. I guess I try too hard to avoid writing to disk when I can.

retzkek 5y ago

Honestly Loki and Promtail are so easy to setup, and jump into not just tailing and searching but incorporating logs into Grafana dashboards (you are using Grafana already to monitor host and service metrics, right?), that they make sense to run for everything but the smallest scale deployment (that is, a single service on a single node).

tstack 5y ago

Sorry to plug lnav again, but if the data volume is not "too much"[1], the latest version of lnav has support for tailing files on remote hosts that are accessible via ssh: https://lnav.org/2021/05/03/tailing-remote-files.html https://lnav.org/2021/05/03/tailing-remote-files.html So, to tail "/path/to/log" on host1 and host2, you'd start lnav with: $ lnav host1:/path/to/log host2:/path/to/log [1] - This is the initial release of this feature, so it probably has some performance issues. I'd be interested to know if it worked or not.

fleur-de-lotus 5y ago
I tried but my timestamp is of the form %Y%m%d.%H%M%S: Do you have an option to just specify the timestamp format? If not, can you give an example of defining own format? Many thanks
- tstack 5y ago
  I'll add this timestamp to the set of defaults for the final v0.10.0 version later today. The documentation for the format files is here: https://docs.lnav.org/en/latest/formats.html https://docs.lnav.org/en/latest/formats.html The "timestamp-format" property in the format definition is an array of the formats to be tried beyond the builtin set. Here are some example formats written by a user: https://github.com/aspiers/lnav-formats https://github.com/aspiers/lnav-formats I think some of them define their own timestamp format.

deleted 5y ago

[deleted]

jedberg 5y ago

Use syslog to send all the logs to one place. Either a third server or just have one send to the other if you can handle the load.