3 ms·

lnav has support for JSON-lines, logfmt, as well as the Bro and W3C Extended Log File formats that are XSV and self-describing. The contents are also accessibl

by tstack 3y ago

lnav has support for JSON-lines, logfmt, as well as the Bro and W3C Extended Log File formats that are XSV and self-describing. The contents are also accessible through SQLite tables. Is there some gap here that you're thinking of?

scottlamb 3y ago
The idea of structured logs is that every place in the code where you define a log message, you can throw in extra attributes (key/value) pairs. As far as I can tell, the lnav feature you describing allows you to define a JSON log format with predefined fields. But if some log message uses an attribute you haven't anticipated in the format definition, there's no way to see it in the pretty-printed output or filter on it in the UI and no ability to see it in the SQLite virtual table. That's why I say lnav doesn't appear to support structured logs. edit: oh, I missed `"hide-extra": false`! That significantly improves things! Still, I don't see a way to access it from the SQLite virtual table. I also don't see something else I might want: a way to have a different "view" for certain log messages; maybe to switch between filtering/viewing particular ones, maybe to just have line-format be conditional based on the detected format. (I guess I can sort of do this based on `module-field`? but I might want it lighter-weight/finer-grained than that.)
- tstack 3y ago
  > But if some log message uses an attribute you haven't anticipated in the format definition, there's no way to see it in the pretty-printed output Properties in the log message that are not in the "line-format" are displayed below the message as key-value pairs. As an example, the bunyan[1] log format[2] has a few standard properties that are used in the "line-format" to form the main message. Then, as shown in this test[3] for this log[4], the remaining properties (even ones not mentioned in the format file) are shown underneath. > or filter on it in the UI Since they are part of the message as mentioned above, they can be filtered on. > and no ability to see it in the SQLite virtual table. The "log_raw_text" column in the table can be used to access the original log message from the file. So, you can use the JSON functions in SQLite to retrieve the value: ;SELECT log_raw_text ->> '$.repository' from bunyan [1] - https://github.com/trentm/node-bunyan https://github.com/trentm/node-bunyan [2] - https://github.com/tstack/lnav/blob/master/src/formats/bunyan_log.json https://github.com/tstack/lnav/blob/master/src/formats/bunya... [3] - https://github.com/tstack/lnav/blob/master/test/expected/test_json_format.sh_6767b91d715338c24c67e928b59c560c84ddf4be.out https://github.com/tstack/lnav/blob/master/test/expected/tes... [4] - https://github.com/tstack/lnav/blob/master/test/logfile_bunyan.0 https://github.com/tstack/lnav/blob/master/test/logfile_buny...
  - scottlamb 3y ago
    Thanks for this. When I switched my project to structured logging, I deleted its lnav log format file and was sad about it. I'll add one back now!
    - tstack 3y ago
      > I also don't see something else I might want: a way to have a different "view" for certain log messages; maybe to switch between filtering/viewing particular ones, maybe to just have line-format be conditional based on the detected format. Have a look at the following comment on an issue that might be similar to what you're thinking of: https://github.com/tstack/lnav/issues/1065#issuecomment-1602971790 https://github.com/tstack/lnav/issues/1065#issuecomment-1602... > I guess I can sort of do this based on `module-field`? but I might want it lighter-weight/finer-grained than that. Unfortunately, the "module-field" does not work for JSON logs at the moment. It's something I should really fix. Ultimately, lnav has existed for almost two decades now and I use it every day. So, it's always seeing improvements. If you're having a problem with it, file an issue on github. I don't always get around quickly to fixing other folks feature requests / issues, but it tends to happen eventually. Thanks.

by tstack 3y ago

scottlamb 3y ago

The idea of structured logs is that every place in the code where you define a log message, you can throw in extra attributes (key/value) pairs. As far as I can tell, the lnav feature you describing allows you to define a JSON log format with predefined fields. But if some log message uses an attribute you haven't anticipated in the format definition, there's no way to see it in the pretty-printed output or filter on it in the UI and no ability to see it in the SQLite virtual table. That's why I say lnav doesn't appear to support structured logs. edit: oh, I missed `"hide-extra": false`! That significantly improves things! Still, I don't see a way to access it from the SQLite virtual table. I also don't see something else I might want: a way to have a different "view" for certain log messages; maybe to switch between filtering/viewing particular ones, maybe to just have line-format be conditional based on the detected format. (I guess I can sort of do this based on `module-field`? but I might want it lighter-weight/finer-grained than that.)

tstack 3y ago
> But if some log message uses an attribute you haven't anticipated in the format definition, there's no way to see it in the pretty-printed output Properties in the log message that are not in the "line-format" are displayed below the message as key-value pairs. As an example, the bunyan[1] log format[2] has a few standard properties that are used in the "line-format" to form the main message. Then, as shown in this test[3] for this log[4], the remaining properties (even ones not mentioned in the format file) are shown underneath. > or filter on it in the UI Since they are part of the message as mentioned above, they can be filtered on. > and no ability to see it in the SQLite virtual table. The "log_raw_text" column in the table can be used to access the original log message from the file. So, you can use the JSON functions in SQLite to retrieve the value: ;SELECT log_raw_text ->> '$.repository' from bunyan [1] - https://github.com/trentm/node-bunyan https://github.com/trentm/node-bunyan [2] - https://github.com/tstack/lnav/blob/master/src/formats/bunyan_log.json https://github.com/tstack/lnav/blob/master/src/formats/bunya... [3] - https://github.com/tstack/lnav/blob/master/test/expected/test_json_format.sh_6767b91d715338c24c67e928b59c560c84ddf4be.out https://github.com/tstack/lnav/blob/master/test/expected/tes... [4] - https://github.com/tstack/lnav/blob/master/test/logfile_bunyan.0 https://github.com/tstack/lnav/blob/master/test/logfile_buny...
- scottlamb 3y ago
  Thanks for this. When I switched my project to structured logging, I deleted its lnav log format file and was sad about it. I'll add one back now!
  - tstack 3y ago
    > I also don't see something else I might want: a way to have a different "view" for certain log messages; maybe to switch between filtering/viewing particular ones, maybe to just have line-format be conditional based on the detected format. Have a look at the following comment on an issue that might be similar to what you're thinking of: https://github.com/tstack/lnav/issues/1065#issuecomment-1602971790 https://github.com/tstack/lnav/issues/1065#issuecomment-1602... > I guess I can sort of do this based on `module-field`? but I might want it lighter-weight/finer-grained than that. Unfortunately, the "module-field" does not work for JSON logs at the moment. It's something I should really fix. Ultimately, lnav has existed for almost two decades now and I use it every day. So, it's always seeing improvements. If you're having a problem with it, file an issue on github. I don't always get around quickly to fixing other folks feature requests / issues, but it tends to happen eventually. Thanks.