4 ms·
Most logging libraries are bad because they're not designed to account for the requirements of a real-time system. What you need: - an asynchronous design whe
by mgaunard 3y ago
Most logging libraries are bad because they're not designed to account for the requirements of a real-time system.
What you need:
- an asynchronous design where both the formatting and the writing to disk can happen on another thread.
- a lockfree buffer using pre-allocated memory. In particular every log record might be of different size.
- the ability to control where/when to drain that buffer, perform formatting and write data to disk (no good library should impose a threading model, needs to be able to cooperatively run with other tasks on one thread)
- an extensible and fast binary serialization/deserialization scheme -- easiest is to rely on simple copy construction as a default.
Most libraries focus on silly things like formatting and filtering which are all trivial problems.
- linkdd 3y ago> Most libraries focus on silly things like formatting and filtering which are all trivial problems. I wanted to print structured data on stdout, without any fuss. All the other libraries that try to tackle the non trivial problems you mention, feel like a bazooka aiming at the ant that is my use case.
- joenot443 3y agoFWIW, in the small desktop app I'm working on now, this library would fit my use case perfectly. Certainly the features mentioned above are powerful, but I think you're right in your belief that plenty of projects don't necessitate such complexity.
- gpderetta 3y agoThey are bad for real time, but not all libraries need to be designed for real time or low latency systems.
- 10000truths 3y agoThose requirements can be met by writing your log data to a pre-allocated memory buffer and flushing the buffer yourself.
- jandrewrogers 3y agoI have often written my own logging implementations for similar reasons. I think it is pretty difficult to turn the set of features into a generic library because it necessarily is opinionated about multiple aspects of the software architecture that uses it. Not all formatting is amenable to being pushed to another thread without introducing undesirable concurrency concerns.
- mgaunard 3y agoThat's why you need an extensible serialization system: to handle data that isn't copyable or whose copies would require memory allocation.
- linkdd 3y agoWhich my library propose with the very simple concept `as_string_trait` that requires the presence of a method `std::string as_string() const`. It's good enough for 100% of my use cases.
- mgaunard 3y agowell clearly that doesn't work, you can't pass it a buffer to write into. This necessary allocates, and I guess it formats on the hot path too.
- linkdd 3y agoWhich is good for 100% of my use cases, since I don't deal with real time systems, or performance intensive applications, or memory/allocation constrained environments. No need to use the bazooka to kill an ant.
- jeffreygoesto 3y agoRight. We did exactly that, for the reasons you wrote, starting from [0] and making it pre-allocated and async. Formatting is as late as possible, even after we store the static and dynamic parts separately in a db. Very versatile (can log to console, json, web frontend, pandas,...) and space and resource efficient. [0] https://youtu.be/FyJI4Z6jD4w?feature=shared https://youtu.be/FyJI4Z6jD4w?feature=shared
- wsve 3y agoI would agree with all of this, _except_ maybe the first bullet. Async can be fine so long as there's a mechanism for flushing all buffers before crashing. Otherwise, there's a high risk that you're gonna miss logging an important event that helps pinpoint why the crash occurred