5 ms·

Will Sargent did a nice talk on logging at Scalacon. My biggest takeaway was that pre-choosing a 'log level' for what to skip is making the decision too early (

Will Sargent did a nice talk on logging at Scalacon. My biggest takeaway was that pre-choosing a 'log level' for what to skip is making the decision too early (it's hard to guess what messages we'll want), and too coarse-grained (we don't want verbose messages when we follow the happy-path, but tricky/exceptional paths may want more logging). His suggestion is to accumulate verbose messages in memory/disk, so we can later decide whether to dump them out (e.g. in a top-level error handler). The talk suggested an SQLite DB, with dynamically-configurable filters for what to keep, etc. but that's a step beyond ;)

adrianmsmith 5y ago
> His suggestion is to accumulate verbose messages in memory/disk, so we can later decide whether to dump them out (e.g. in a top-level error handler). Interesting; I've definitely seen that with unit test frameworks. Where in addition to assertEquals(...) there was an annotate(...) method which just took a string, and was stored for that unit test. If the unit test eventually failed, all the previous annotations for that unit test were printed, otherwise they weren't. https://metacpan.org/pod/Test::Unit::TestCase#annotate-(MESSAGE) https://metacpan.org/pod/Test::Unit::TestCase#annotate-(MESS...
worik 5y ago
I really do not think that logs belong in any sort of database. Perhaps there is something I have not thought of? Structured logs (of course structured) in a simple text file. Rotate the files if your antiquated system cannot handle big files or if you wildly successful crypto currency scam generates many many gigabytes a time period..... `grep` is your friend!
- chriswarbo 5y ago
  I like the idea of a simple SQLite DB, but agree it's a bit overkill. In particular, it increases the barrier to writing a log message (what structure/keys/etc. to choose?), which disincentivises logging. The accumulate-then-dump idea doesn't require any database though; just an in-memory buffer (e.g. a linked-list). For example, we can have all ERROR/WARNING/INFO messages output immediately, as is current practice. The difference would be for DEBUG messages: - Common practice is that if DEBUG is set then we'll output DEBUG messages as they're encountered; if DEBUG is not set then we'll skip over them. - The suggestion is that if DEBUG is not set, then instead of skipping messages the logger would put them in a buffer. If an unrecoverable error occurs, we can put a call like `logger.dumpDebug()` in the handler, so all those DEBUG messages will appear in the logs to help us diagnose the error. If no error occurs, the buffer just gets discarded. This obvious takes more memory, but fits well with a request/response setup like a server. Long-running applications/games could either use a ring buffer, or identify points where it makes sense to empty the buffer.

adrianmsmith 5y ago

> His suggestion is to accumulate verbose messages in memory/disk, so we can later decide whether to dump them out (e.g. in a top-level error handler). Interesting; I've definitely seen that with unit test frameworks. Where in addition to assertEquals(...) there was an annotate(...) method which just took a string, and was stored for that unit test. If the unit test eventually failed, all the previous annotations for that unit test were printed, otherwise they weren't. https://metacpan.org/pod/Test::Unit::TestCase#annotate-(MESSAGE) https://metacpan.org/pod/Test::Unit::TestCase#annotate-(MESS...

worik 5y ago

I really do not think that logs belong in any sort of database. Perhaps there is something I have not thought of? Structured logs (of course structured) in a simple text file. Rotate the files if your antiquated system cannot handle big files or if you wildly successful crypto currency scam generates many many gigabytes a time period..... `grep` is your friend!

chriswarbo 5y ago
I like the idea of a simple SQLite DB, but agree it's a bit overkill. In particular, it increases the barrier to writing a log message (what structure/keys/etc. to choose?), which disincentivises logging. The accumulate-then-dump idea doesn't require any database though; just an in-memory buffer (e.g. a linked-list). For example, we can have all ERROR/WARNING/INFO messages output immediately, as is current practice. The difference would be for DEBUG messages: - Common practice is that if DEBUG is set then we'll output DEBUG messages as they're encountered; if DEBUG is not set then we'll skip over them. - The suggestion is that if DEBUG is not set, then instead of skipping messages the logger would put them in a buffer. If an unrecoverable error occurs, we can put a call like `logger.dumpDebug()` in the handler, so all those DEBUG messages will appear in the logs to help us diagnose the error. If no error occurs, the buffer just gets discarded. This obvious takes more memory, but fits well with a request/response setup like a server. Long-running applications/games could either use a ring buffer, or identify points where it makes sense to empty the buffer.