4 ms·

The rookie mistake a lot of admins do is never creating a partition for /var/log , I lost count how many times servers went into a weird mode when the root is g

The rookie mistake a lot of admins do is never creating a partition for /var/log , I lost count how many times servers went into a weird mode when the root is getting filled to 100%

est 7y ago
I always rm the dir inside /var/log and ln them back from another disk. Because I am too lazy to change default config sparsely located on /etc of many installed programs.
- Galanwe 7y ago
  Most of the programs should by default use syslog, so you should not really have to configure much of them except enabling syslog output. Then you just have to configure your syslog implementation to write wherever you desire, rotate the files, or forward the messages to other machines for storing, etc.
  - majewsky 7y ago
    > Most of the programs should by default use syslog Is that still true today? Docker wants you to log to stdout, so that's what most newer applications do. systemd also wants you to log to stdout, and will redirect stdout to journald/syslog automatically. In fact, an application that only logs to syslog can turn into a minor headache when you want to dockerize it. Which is why stuff like https://github.com/sapcc/syslog-stdout https://github.com/sapcc/syslog-stdout exists. I'm having a related problem with another project that I'm working on where I wrap an OpenLDAP server. It would be much easier to properly wrap it if OpenLDAP would just log to stdout instead of bypassing me and going for the syslog. Maybe at some point I'll set up a separate mount namespace for it to pass a /dev/syslog shim into it. But this shows that the log-to-stdout pattern is much more Unix-y because it composes better.
    - defanor 7y ago
      > Is that still true today? I think it is. > Docker wants you to log to stdout, so that's what most newer applications do. According to Debian popularity contest [1], which matches my observations, Docker itself isn't a particularly common package to be found in a system. > systemd also wants you to log to stdout While that's an option, there is sd-journal(3), which allows proper logging with priorities and custom fields. [1] https://popcon.debian.org/by_inst https://popcon.debian.org/by_inst Edit: Perhaps worth mentioning that even with systemd and logging into stdout, syslogd (and maybe journald) configuration should be sufficient to sort out the log files, as mentioned in the grandparent comment.
    - imtringued 7y ago
      If your dockerized application logs to a file you can always symlink that file to /dev/stdout as a last resort.
hinkley 7y ago
I broke a machine today by deploying a docker image to it. I couldn’t even ssh in because the disk was too full. Which probably means all of those machines have the same fucked up bad configuration. If that team had something you could mistake for humility, it wouldn’t be so bad.
lousken 7y ago
one word: logrotate

The rookie mistake a lot of admins do is never creating a partition for /var/log , I lost count how many times servers went into a weird mode when the root is getting filled to 100%

est 7y ago

I always rm the dir inside /var/log and ln them back from another disk. Because I am too lazy to change default config sparsely located on /etc of many installed programs.

Galanwe 7y ago
Most of the programs should by default use syslog, so you should not really have to configure much of them except enabling syslog output. Then you just have to configure your syslog implementation to write wherever you desire, rotate the files, or forward the messages to other machines for storing, etc.
- majewsky 7y ago
  > Most of the programs should by default use syslog Is that still true today? Docker wants you to log to stdout, so that's what most newer applications do. systemd also wants you to log to stdout, and will redirect stdout to journald/syslog automatically. In fact, an application that only logs to syslog can turn into a minor headache when you want to dockerize it. Which is why stuff like https://github.com/sapcc/syslog-stdout https://github.com/sapcc/syslog-stdout exists. I'm having a related problem with another project that I'm working on where I wrap an OpenLDAP server. It would be much easier to properly wrap it if OpenLDAP would just log to stdout instead of bypassing me and going for the syslog. Maybe at some point I'll set up a separate mount namespace for it to pass a /dev/syslog shim into it. But this shows that the log-to-stdout pattern is much more Unix-y because it composes better.
  - defanor 7y ago
    > Is that still true today? I think it is. > Docker wants you to log to stdout, so that's what most newer applications do. According to Debian popularity contest [1], which matches my observations, Docker itself isn't a particularly common package to be found in a system. > systemd also wants you to log to stdout While that's an option, there is sd-journal(3), which allows proper logging with priorities and custom fields. [1] https://popcon.debian.org/by_inst https://popcon.debian.org/by_inst Edit: Perhaps worth mentioning that even with systemd and logging into stdout, syslogd (and maybe journald) configuration should be sufficient to sort out the log files, as mentioned in the grandparent comment.
  - imtringued 7y ago
    If your dockerized application logs to a file you can always symlink that file to /dev/stdout as a last resort.

hinkley 7y ago

I broke a machine today by deploying a docker image to it. I couldn’t even ssh in because the disk was too full. Which probably means all of those machines have the same fucked up bad configuration. If that team had something you could mistake for humility, it wouldn’t be so bad.

lousken 7y ago

one word: logrotate