3 ms·
I've been solving similar problem to decouple python daemon logging IO from event loop in order to make logging completely non-blocking. I'm pretty sure this pa
by Snawoot 6y ago
I've been solving similar problem to decouple python daemon logging IO from event loop in order to make logging completely non-blocking. I'm pretty sure this particular case must be most popular file IO case for async applications.
In my case it wasn't important to open log asynchronously, but it was critical to have async writes.
It turns out, depths of Python stdlib hide logging.handlers.QueueHandler and logging.handlers.QueueListener which allow to split log message handler into separate thread from log message emitters, making them communicate via synchronized queue.
Final solution is quite compact:
https://github.com/Snawoot/postfix-mta-sts-resolver/blob/589a59a90cf3f53148f9f3814ccd48cd92a9538f/postfix_mta_sts_resolver/utils.py#L26-L57 https://github.com/Snawoot/postfix-mta-sts-resolver/blob/589...
Usage example:
https://github.com/Snawoot/postfix-mta-sts-resolver/blob/589a59a90cf3f53148f9f3814ccd48cd92a9538f/postfix_mta_sts_resolver/daemon.py#L101-L106 https://github.com/Snawoot/postfix-mta-sts-resolver/blob/589...
It is convenient that you can still use logging as usual, without any
async obligations. You just have logging function which works in a
fire-and-forget fashion.
> Asynchronous reads and writes would require all new APIs with different coloring. You’d need an aprint() to complement print(), and so on, each returning an awaitable to be awaited.
> This is one of the unfortunate downsides of async/await. I strongly prefer conventional, preemptive concurrency, but we don’t always have that luxury.
I was sceptical about Golang for a long time, but recently I've started practicing it. It's designed with concurrency in mind from very beginning and goroutines have advantages of both cooperative and preemptive multitasking. And also it doesn't have GIL, so your goroutines will use all available CPUs without any additional effort from your side.
From programmers point of view, coding with goroutines looks exactly like coding with use of pthreads (even simpler, in fact). So, there are no function "coloring", no need in sync/async library flavors and so on. But these "threads" also have cost close to cost of python coroutines. It sounds too good to be true, but I've rewritten bunch of my Python/asyncio code in Go and I can say it's true. For this reason (and many others I discovered later) I quit using Python and asyncio for network applications in favor of Go.
- nly 6y agoFor anyone looking for async logging in C++ I can highly recommend spdlog: https://github.com/gabime/spdlog https://github.com/gabime/spdlog