4 ms·
Am I the only person that doesn't want logging to block or buffer? Streams are great until you realize that when the thing that you are piping stdout to blocks
by jrockway 16y ago
Am I the only person that doesn't want logging to block or buffer? Streams are great until you realize that when the thing that you are piping stdout to blocks then your program goes to sleep too... all because something couldn't process your debug logs? Files have the same problem... no space left on device, so all your apps have to crash!?
Using non-blocking IO is a solution, but then when the fd is unwritable for a while, you buffer data and your machine runs out of memory. So also not good.
I'm a big fan of the strategy Varnish uses. It mmaps a block of memory, and then treats it as a ring buffer. A separate program can connect to this and read the logs to a file or a pipe, but if that program fails, you don't stop serving web pages. As long as allocated memory doesn't randomly disappear and the Universe doesn't implode, logging adds no overhead or risk of failure to your app.
Logs are essential, but I don't necessarily want a failure in the logging system to break my app. When you use files or pipes, you run this risk.
- cachemoney 16y agoYou can log to mongodb capped collections. You get a semi-structured ring buffer with fast writes. I'm not a huge mongo fanboi, but this is one situation I've seen it excel in.
- cparedes 16y agoI believe syslog-ng is non-blocking. You don't have to reinvent the wheel.
- wiredfool 16y agoSyslog had blocking and non blocking writes, depending on the severity of the log. I suspect that it's still that way in syslog-ng,but I haven't looked at it recently.
- StavrosK 16y agoThis is not entirely on topic, but I'm still blown away from the UDP approach the other day. Send a UDP packet to your log collector and forget about it. It's lightning fast, the recipient can drop packets if it can't handle the load and your program will never be any the wiser.
- spc476 16y agoLast year our upstream provider broke TCP (they upgraded a router/firewall box and it wasn't supposed to affect us, but it dropped asymetrically routed TCP traffic) but UDP packets still got out, and thanks to seeing UDP syslog traffic making it out helped us diagnose the issue. Using the "more reliable TCP method" (rsyslogd supports this) would have made the issue harder to track down.
- brown9-2 16y agostill blown away from the UDP approach the other day what is this in reference to?
- wiredfool 16y agoStatsd/graphite, I think.
- jrockway 16y agoUh, and UNIX syslog since like 1970?
- wiredfool 16y agoYeah, but no one did a shiny post about syslog a couple of days ago.
- StavrosK 16y agoSyslog is more about human-readable logging, but you're right, it's more relevant in this context. The graphite approach was about fast aggregation of samples (app response times, load, etc) and the graphing of these.
- haberman 16y ago> Am I the only person that doesn't want logging to block or buffer? If it's an error that you're logging the moment before you crash, then you probably want it to block. At Google LOG(ERROR) blocks but LOG(INFO) doesn't, for this reason.
- jrockway 16y agoSmart! I think it might finally be time for me to send my resume in the direction of Google ;)
- rbanffy 16y agoFor this very same reason you should have separate ring buffers for different types of log. I would prefer to overwrite INFO entries but not access entries - because we have to frequently deal with court orders that require us to provide access logs. It's relatively rare to use production server error logs - we only need them when things go very wrong and we try to avoid that.