11 ms·
Discussion: structured, leveled logging
- motiejus 4y agoIf we ignore the proposal, this gives at least an awesome overview of the state-of-the-art of logging in go. That was the best part. Thanks for the heads up!
- egnehots 4y agoPrior Work - The existing log package has been in the standard library since the release of Go 1 in March 2012. It provides formatted logging, but not structured logging or levels. - Logrus, one of the first structured logging packages, showed how an API could add structure while preserving the formatted printing of the log package. It uses maps to hold key-value pairs, which is relatively inefficient. - Zap grew out of Uber's frustration with the slow log times of their high-performance servers. It showed how a logger that avoided allocations could be very fast. - zerolog reduced allocations even further, but at the cost of reducing the flexibility of the logging backend. - All the above loggers include named levels along with key-value pairs. Logr and Google's own glog use integer verbosities instead of named levels, providing a more fine-grained approach to filtering high-detail logs. Other popular logging packages are Go-kit's log, HashiCorp's hclog, and klog.
- thechao 4y agoAll the sqlite love, here, recently... but none on logging? Using a `:memory:` SQLite DB, combined with the backup API is perfect for one-shot application (non-server processes). The formatting I do is cleverly composed SQL queries.
- srcreigh 4y agoAre you still talking about logging? Anyways, would love to hear more about your clever queries, even if it's OT.
- thechao 4y agoYep. The "style" of logger is in C, and looks (roughly) like this: Logger* OpenLog(char*); Table* MakeLog(Logger*, unsigned level, unsigned N, char* colNames, ColType *colTypes); LOG(Table, Level, ...); The branch to select log-level is trivial compared to the append cost. There's a few "backends" built in to the API: (1) a text-streaming version that just writes a text log; (2) a "bucketing" version that boils the calls to `MakeLog` down to accumulating buckets (with optional stack tracing); and, (3) journaling to a SQLite DB (with optional stack tracing).
- srcreigh 4y agoDo you use glibc backtraces for stack tracing? I don't use C much, i guess that may be standard. or something else? Also, can you describe the queries you run against this SQLite? Is it interactive use or do you have any jobs or charts that are powered by it? So very interested in this whole approach - if you have any posts or links to read more happy to see em!
- wsargent 4y agoSame! https://github.com/tersesystems/blacklite/ https://github.com/tersesystems/blacklite/
- svnpenn 4y agoEvery time I come across a logging package, I don't understand what the point of them are. Finally in the linked issue, I see this: > For many programs that aren't run directly by person, like servers, logging is the main way for developers to observe the detailed behavior of the system, and often the first place they go to debug it. and it makes sense. I don't deal with servers at all, other than a couple I have made just to host some basic sites or programs locally. I think Go team might be overlooking the fact that a big chunk of the userbase doesnt understand, and has no need for a logging package. Couple that with the giant API proposed, along with a poor design decision (kvs ...any), and the large amount of bikeshedding that already happened in this issue, I really hope they think hard before just shoving this into "std".
- atsjie 4y agoThe majority of Golang devs actually do work on web services, API's, or libs/frameworks that would benefit from a standardized structured logging interface. Source: https://go.dev/blog/survey2022-q2-results https://go.dev/blog/survey2022-q2-results (scroll almost to the bottom) For the audience of this proposal it's probably fairly evident what the advantages of standardization would be. Also you argue quite strongly about the poor API design decisions being made; while you similarly admit not having worked with such tools or with web services in Golang general. That's odd.
- svnpenn 4y agoI don't work with servers much, but I do have extensive experience in Go. I know a bad API when I see one. strings.Replacer [1] uses the same syntax, and several others have commented in the discussion about it being a bad idea. 1. https://godocs.io/strings#Replacer https://godocs.io/strings#Replacer
- xani_ 4y agoFrom what I've skimmed the most problems seem to be "golang's type system won't allow expressing all of that on compile time so it will be costly" and I don't think that can be sensibly fixed in logging package > I know a bad API when I see one. strings.Replacer [1] uses the same syntax, and several others have commented in the discussion about it being a bad idea. What's wrong with it ? Never used it. But I think the complaints has been about wasting time for runtime type handling and Replacer doesn't have that problem.
- GauntletWizard 4y agoLevels are a great abstraction for debug logs. Debug logs are often abused for much more than their reasonable purpose. Audit logs are different from debug logs, and it's pretty common for people to not think about the reasons why. Audit logs are records, and they don't have levels; they might have metadata, but you need to capture all of them, reliably. Debug logs are ephemeral and if you miss some of them you shouldn't care. The problem with structured logging libraries is that they try to do both. There's some in common, but the mechanisms of output should be really different - Debug logs in a circular buffer in RAM makes a lot of sense, but you can't do that for Audit logs. The current approach seems to be "Write them all to the same place, then filter them out", and that only adds more latency in which to lose your audit logs.
- tptacek 4y agoThis appears to be why there's a separation of concerns between "handlers" and "loggers".
- GauntletWizard 4y agoThat's exactly what I'm railing against. You shouldn't be pushing all these things into the same stream and then separating them out again. You may want to make it easy to generate a debug log when you generate an alert. You shouldn't do that by writing a log. You should have a very different set of parameters for If you're a bank, your login "Logs" shouldn't just be an annotated web request; you should have an explicit "AuthenticationSuccess" type in a non-ephemeral data store, and it should contain more than just "301 /auth 1.2.3.4", it should include information about the factors used to make that decision. Semi-structured logs don't have that, but will be abused to provide that. It should be a type system error to create an AuthSuccess without all that, not a set of default parameters to a generic logging call. Structured logging is great for moving fast, and I use it all the time to make it easy to filter through my debug logs, but levels are an abstraction in debug logs - and handlers are completely an anti-pattern
- tptacek 4y ago
- glenjamin 4y agoI’m increasingly of the opinion that we should stop using “logging” libraries, and instead we should produce telemetry via events using something like OpenTelemetry’s trace API. These traces can be rendered as strings for people who like their logs in files
- rcarmo 4y agoMany tracing libraries get their data off logging libraries. Logging doesn't always mean "whoa, let's stop everything and write to a file", it is often a matter of buffering data or increasing counters until data can be sent out in various fashions. Log files are just one of those.
- glenjamin 4y agoThe key difference is my experience is not about destination - backends are as flexible as you say One key difference is having an event name + properties, rather than a human-readable description and maybe some fields if you’re lucky Another key difference is wrapping units of work rather than just scattering log lines in places you think need them somewhat arbitrarily
- rcarmo 4y agoMost decent logging frameworks allow for structured output. That change came about roughly when NewRelic became "the thing" to use.
- jayd16 4y ago>One key difference is having an event name + properties [...] Another key difference is wrapping units of work [...] Modern "enterprise" logging libraries already work this way, don't they?
- markmark 4y agoYou probably need to look into modern logging if you think that's how logging still works.
- 4y ago
- latchkey 4y agoWhile we can debate all day long what should and should not be part of the standard lib (bike shed), I just ignore all of this and happily use zerolog. Problem solved, one dependency, it just works. Back to writing code.
- tptacek 4y agoThe proposal acknowledges that. Part of the point of working this out is that it creates a common interface so that your zerolog-logging code plays nice with someone else's hclog-logging code.
- oezi 4y agoUnless the new generic logging API is designed badly so that certain functionalities become hard in the future because they deviate. For instance when Rails did standardize ActiveJob (background tasks) they didn't include a way in the API to query the current length of a queue of background tasks. Because everyone is now only trying to conform to the API accessing such information is now hard in many of the implementations.
- latchkey 4y agoA long time ago, I was part of JSR-47. I wonder if anything can be learned from that process (which was a bit of a mess).
- 4y ago
- didip 4y agoIf Zap and Zerolog can agree on a common interface, I think the rest of Go world would follow.
- xani_ 4y agoSeriously, Zap nails most of what I want from logger and the remaining few % isn't exactly hard to add on top. Could maybe get a bit of ergonomics pass but that's it
- javier2 4y agoI have not used Zap, but I really like the Zerolog API which I have used extensively. I dont have any religious fight with Zap vs Zerolog, I just happened to use Zerolog first.
- reggieband 4y agoI see some asking for stack support in the logs, and that is something I would be looking for. Often when an error happens on a server it might be inside some generic utility. At least with a stack, you might realize that all of the calls to that utility function are coming from the same location a few steps up the stack. In my experience that can significantly speed up identifying the root cause of issues. I'm not sure I like the way they are handling attributes either. One pattern I strongly discourage on projects I work on is formatting data into the log message. When I've had access to decent log aggregation tools (e.g. Splunk, Datadog) I like to generate reports on error messages. That is usually possible with regex magic but in my experience it is much much cleaner to have a static unchanging message for all messages. I then store all additional details in a separate part of the log line. I tend to prefer JSON for those details since it is well supported and even parsed by default by many tools. Another benefit of JSON for those additional details is that it is supports arrays and hash-tables. The attribute system proposed here doesn't seem to have that same hierarchical attribute support. IMO, unified, clean and consistent logging is one of the most underrated techniques for maintaining long running services.
- suralind 4y agoThis is what annoys me in my current job - every service uses unstructured logging. We use a couple popular loggers in Go and still, people put values in messages rather than just make the message static and put all variables into, well, variables.
- XorNot 4y agoThese days I kind of wish we didn't do text logging at all. There's essentially two types of logs: fixed messages, and values. A binary logging protocol can make this definition explicit and more efficient: you're either emitting a token for the fixed message in your applications format, or you're emitting values...and a fixed token for the format string, which can be parsed and reconstructed later. Our applications don't need to be doing this sort of text formatting in them at all, and we need the developer interface to make explicit what's happening and what's important.
- 4y ago
- s-xyz 4y agoGreat initiative and well structured problem. Dealt with something similar before, not as detailed as this, but with focus on logging metrics for the purpose of analytics and performance measurement (yes I am aware a db would have been an option too for the analytical part - long story to explain the rationale that I’ll leave out here).