4 ms·
Relevant comment: https://news.ycombinator.com/item?id=6799328 https://news.ycombinator.com/item?id=6799328 " CloudFlare generates 50gb/s of logs globally and
by radq 13y ago
Relevant comment: https://news.ycombinator.com/item?id=6799328 https://news.ycombinator.com/item?id=6799328
" CloudFlare generates 50gb/s of logs globally and have handled collecting this volume in two ways. Historically the logs are sent to a local syslog-ng through the use of a PIPE and the forwarded to central logger. This can be done with nginx with no patches by just treating the PIPE as file. Just make sure you do a little buffering inside nginx.
access_log /dev/nginx_access log_format_name buffer=64k flush=10s;
Since this is a pipe there is still some blocking IO, but no worse off then writing to local file."
- otterley 13y agoIf the pipe consumer (i.e. your syslog server) stalls, then it will be much worse than writing to a local file, as it could cause a denial of service. I would still configure nginx to write to a file and have the consumer tail it to avoid this situation. Properly-configured log rotation can keep the file size within reasonable bounds.
- jsmeaton 13y agoRelated question - how do you configure logrotate to honour size immediately? In my (limited) experience, logrotate runs once overnight. If you have a log file that expands by 1gb per hour, and you want to rotate it every 500mb, how do you instruct logrotate to handle it? Do you just crontab logrotate for short intervals, and will that trigger "daily" rotations each time for other configurations?
- otterley 13y agoThere's no reason why you can't run logrotate more than once a day. You can run a separate instance every 15 minutes if you like, with a separate configuration file that only rotates the short-lived logs.
- jsmeaton 13y agoOk that's the (easy) thing I was missing. Instead of using the main logrotate.conf (that includes logrotate.d/*), use a fully separate logrotate.conf. Thanks.
- asdasf 13y ago>If the pipe consumer (i.e. your syslog server) stalls, then it will be much worse than writing to a local file, as it could cause a denial of service. But how likely is that to happen in any circumstance that doesn't also bring down nginx anyways? Obviously in the case of a large setup like that they don't care, it just gets removed from the pool regardless of why it is failing.
- staunch 13y agoExtremely likely. I've seen the rsyslog daemon die randomly many times, for example. Whereas in 6+ years of using Nginx I've never seen it fall over a single time. Web serving is usually a primary purpose for a server. You do not want your web site to be down just because the machine's syslog daemon died.
- voltagex_ 13y agoIsn't this a reason to replace the syslog daemon rather than the webserver?
- staunch 13y agoIt's a reason to keep things as loosely coupled as possible. And when I say I've seen rsyslog crash many times, I'm talking about over the course of years and thousands of servers.
- asdasf 13y agoOur notions of "extremely likely" differ then. I have never had syslogd crash. If you have, you should be switching to a different syslogd.