46 ms·

Which has better odds, the smoldering remains having actually flushed the write to disk or the flushed to the upstream log aggregator?

• hughrr 5y ago
  From experience of both, the smouldering remains seems to win out by a fair margin. The aggregator may have some smoke signals in it but that’s about it. Most of the issues I’ve dealt with shaft the network before the filesystem.
  • fullstop 5y ago
    Outside of the "cloud world" most of my failures have been the result of disk failure. In those cases, the logs definitely did not get written to disk, but they were flushed to graylog.
    • 0xbadcafebee 5y ago
      I can't remember the name of it anymore, but we used to use a user-space daemon to collect logs from applications over various methods (sockets pipes files etc) and it would store them in a ring buffer until they could be flushed to disk or network. If neither happened, you could attach to the host somehow and get the logs out of memory, watch them on a terminal screen, etc. Opening an EC2 console in AWS is still a simple and reliable way to find out what's going wrong with your instance. Wouldn't be possible if we didn't already have the convention to have the kernel, syslog, etc print to tty1.