8 ms·

My biggest quality of life improvement for understanding logs has been lnav (https://lnav.org/ https://lnav.org/) -- does everything mentioned in this post in a

My biggest quality of life improvement for understanding logs has been lnav (https://lnav.org/ https://lnav.org/) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

clumsysmurf 4y ago
Interesting! Do you know how it compares to multitail? (https://www.vanheusden.com/multitail/ https://www.vanheusden.com/multitail/) They look very similar.
- tstack 4y ago
  I don't think multitail really understands logs like lnav does, it's just following the last lines in the file. For example, if you try to follow multiple files in multitail like so: $ multitail /var/log/install.log -I /var/log/system.log You get a view with the tail from one file followed by the tail from the other, they are not collated by timestamp. In contrast, if you do the same thing in lnav: $ lnav /var/log/install.log /var/log/system.log You will get a single view with all of the log messages from both files and they're sorted by their timestamps. Here is all of what lnav is doing: * Monitoring files/directories for updates * Decompressed files/archives * Detected the log format for each file * Created SQLite vtables that provide access to log messages * Built an index of all the log messages in all the files so you can jump to a certain point in time or to the next/previous error message. * Display all log messages with syntax highlighting
  - aargh_aargh 4y ago
    I migrated from multitail to lnav. Turned out to be a no-brainer. I second the above, just one pain point with multitail to add. I often page/search/filter in the scrollback buffer (I typoed "bugger" - Freudian slip?) and in multitail the scrollback is a separate window with a frame and everything, which is a pain (copying whole lines using mouse includes the frame, ugh). The filtering/searching being a separate pain. One thing I used in multitail and not sure if I migrated wholly to lnav was log file syntax highlighting using regexes.
TeMPOraL 4y ago
Thanks for linking to this! Until now I thought logview.el[0] is the bee's knees, but now I can feel feature envy set in. There are some seriously powerful ideas listed on the lnav page, and it's also the first time I saw SQLite virtual tables used in the wild. -- [0] - https://github.com/doublep/logview https://github.com/doublep/logview
oefrha 4y ago
Huh, I almost posted a duplicate recommendation. My only complaint with lnav was that it had to be built from source on Linux and the build was frigging huge. Apparently they have a pre-compiled linux-musl binary now.
- amarshall 4y ago
  > that it had to be built from source on Linux Seems to be packaged quite widely https://repology.org/project/lnav/versions https://repology.org/project/lnav/versions
- joshka 4y ago
  I used nix when I had to make lnav work on some machines where this would have been difficult.
francislavoie 4y ago
At my current company, we're using pdsh with tail -f piped into sed to follow logs from multiple remote servers at the same time, label them and colorize them. Works okay. Decent solution without needing to install much other software. Not my favourite because it doesn't deal well with SSH sessions timing out and leaves some tail -f processes hanging, and some other quirks. But out of laziness and risk of breaking things in prod, we haven't tried experimenting with much else.
- UI_at_80x24 4y ago
  Run that in a tmux session. Problem solved.
- tstack 4y ago
  lnav has some support for tailing files on remote hosts -- https://lnav.org/2021/05/03/tailing-remote-files.html https://lnav.org/2021/05/03/tailing-remote-files.html It works by uploading a stub that is an "actually portable executable" to the remote over ssh. lnav then talks to the stub to sync files back to the local host. I don't know the scale of the hosts or files, but if they're not too big, it might work fine.
chrisweekly 4y ago
Yes! lnav (https://lnav.org https://lnav.org) is phenomenal. Embedded SQLite... easily scriptable... OOB log formats galore, or define your own... it's a mini ETL powertool that scales to at least a few million rows and runs in your terminal. Maintainer's a friendly dude, too.
hermitcrab 4y ago
Our Easy Data Transform software is intended for data wrangling and desktop ETL. But it has lots of features useful for browsing and manipulating log files, including: * powerful filtering (including with regex) * smooth scrolling of millions of rows of data * support for csv, text, xml, json, Excel and other formats * available for Window and Mac
heinrichhartman 4y ago
Looks great, but I struggle to get this working with the two main log sources on my system: 1) systemd logs (binary data stored at /var/log/journal) 2) docker logs (from various containers running on the host) Any hints on getting this working? I don't see this covered in the docs (https://docs.lnav.org/en/latest/formats.html https://docs.lnav.org/en/latest/formats.html).
- funcDropShadow 4y ago
  for systemd logs this works for me journalctl -o json | lnav
dj_gitmo 4y ago
I love lnav and use it constantly, but it crashes a lot. I do wish there was something like lnav that was a little simpler to use, and written in a more resilient way that crashed less. I can cut lnav some slack for the crashes because identifying and parsing arbitrary log formats seems like a messy problem. Still it shouldn't crash 1/3rd of the time I use it.
- tstack 4y ago
  Sorry for the crashes :( I've been trying to improve it's internals more than adding features as of late. If you haven't already, please file bugs on github and/or submit crash logs to the mailing list. (I've taken a break from working on it lately. So, if you've done that and I haven't gotten back, I apologize.)
  - dj_gitmo 4y ago
    No worries. It's a great tool and I'd be very happy to make reports and share crash logs.
joshka 4y ago
Came here to say the same - massive props to the lnav devs.

clumsysmurf 4y ago

Interesting! Do you know how it compares to multitail? (https://www.vanheusden.com/multitail/ https://www.vanheusden.com/multitail/) They look very similar.

tstack 4y ago
I don't think multitail really understands logs like lnav does, it's just following the last lines in the file. For example, if you try to follow multiple files in multitail like so: $ multitail /var/log/install.log -I /var/log/system.log You get a view with the tail from one file followed by the tail from the other, they are not collated by timestamp. In contrast, if you do the same thing in lnav: $ lnav /var/log/install.log /var/log/system.log You will get a single view with all of the log messages from both files and they're sorted by their timestamps. Here is all of what lnav is doing: * Monitoring files/directories for updates * Decompressed files/archives * Detected the log format for each file * Created SQLite vtables that provide access to log messages * Built an index of all the log messages in all the files so you can jump to a certain point in time or to the next/previous error message. * Display all log messages with syntax highlighting
- aargh_aargh 4y ago
  I migrated from multitail to lnav. Turned out to be a no-brainer. I second the above, just one pain point with multitail to add. I often page/search/filter in the scrollback buffer (I typoed "bugger" - Freudian slip?) and in multitail the scrollback is a separate window with a frame and everything, which is a pain (copying whole lines using mouse includes the frame, ugh). The filtering/searching being a separate pain. One thing I used in multitail and not sure if I migrated wholly to lnav was log file syntax highlighting using regexes.

TeMPOraL 4y ago

Thanks for linking to this! Until now I thought logview.el[0] is the bee's knees, but now I can feel feature envy set in. There are some seriously powerful ideas listed on the lnav page, and it's also the first time I saw SQLite virtual tables used in the wild. -- [0] - https://github.com/doublep/logview https://github.com/doublep/logview

oefrha 4y ago

Huh, I almost posted a duplicate recommendation. My only complaint with lnav was that it had to be built from source on Linux and the build was frigging huge. Apparently they have a pre-compiled linux-musl binary now.

amarshall 4y ago
> that it had to be built from source on Linux Seems to be packaged quite widely https://repology.org/project/lnav/versions https://repology.org/project/lnav/versions
joshka 4y ago
I used nix when I had to make lnav work on some machines where this would have been difficult.

francislavoie 4y ago

At my current company, we're using pdsh with tail -f piped into sed to follow logs from multiple remote servers at the same time, label them and colorize them. Works okay. Decent solution without needing to install much other software. Not my favourite because it doesn't deal well with SSH sessions timing out and leaves some tail -f processes hanging, and some other quirks. But out of laziness and risk of breaking things in prod, we haven't tried experimenting with much else.

UI_at_80x24 4y ago
Run that in a tmux session. Problem solved.
tstack 4y ago
lnav has some support for tailing files on remote hosts -- https://lnav.org/2021/05/03/tailing-remote-files.html https://lnav.org/2021/05/03/tailing-remote-files.html It works by uploading a stub that is an "actually portable executable" to the remote over ssh. lnav then talks to the stub to sync files back to the local host. I don't know the scale of the hosts or files, but if they're not too big, it might work fine.

chrisweekly 4y ago

Yes! lnav (https://lnav.org https://lnav.org) is phenomenal. Embedded SQLite... easily scriptable... OOB log formats galore, or define your own... it's a mini ETL powertool that scales to at least a few million rows and runs in your terminal. Maintainer's a friendly dude, too.

hermitcrab 4y ago

Our Easy Data Transform software is intended for data wrangling and desktop ETL. But it has lots of features useful for browsing and manipulating log files, including: * powerful filtering (including with regex) * smooth scrolling of millions of rows of data * support for csv, text, xml, json, Excel and other formats * available for Window and Mac

heinrichhartman 4y ago

Looks great, but I struggle to get this working with the two main log sources on my system: 1) systemd logs (binary data stored at /var/log/journal) 2) docker logs (from various containers running on the host) Any hints on getting this working? I don't see this covered in the docs (https://docs.lnav.org/en/latest/formats.html https://docs.lnav.org/en/latest/formats.html).

funcDropShadow 4y ago
for systemd logs this works for me journalctl -o json | lnav

dj_gitmo 4y ago

I love lnav and use it constantly, but it crashes a lot. I do wish there was something like lnav that was a little simpler to use, and written in a more resilient way that crashed less. I can cut lnav some slack for the crashes because identifying and parsing arbitrary log formats seems like a messy problem. Still it shouldn't crash 1/3rd of the time I use it.

tstack 4y ago
Sorry for the crashes :( I've been trying to improve it's internals more than adding features as of late. If you haven't already, please file bugs on github and/or submit crash logs to the mailing list. (I've taken a break from working on it lately. So, if you've done that and I haven't gotten back, I apologize.)
- dj_gitmo 4y ago
  No worries. It's a great tool and I'd be very happy to make reports and share crash logs.

joshka 4y ago

Came here to say the same - massive props to the lnav devs.