3 ms·

My server went down unexplainably and I manage to bring it back up. I'll be tailing every single log file I can find until I get a clue of what the problem is.

• otoolep 5y ago
  What is about your deployment or setup that you don't use your log search system instead? Assuming you have one? You could search all your logs files at once (for errors, for strack traces), instead of manually tailing each one, one after the other. Is that you don't even know what to search for?
  • NBJack 5y ago
    You are making assumptions that the log upload system is working in the first place. It's just another moving part in the equation. Since logging normally shouldn't affect production traffic latency, you usually want to offload responsibility to a separate process/sidecar/etc. That requires intermediate storage as logs are generated, which can fill up (disk or ram, it's all finite). Since you don't want just any person sending logs for processing, you now have to ensure this separate process is authorized to upload data, which means you probably have another security token in play. The search system must index the data in near real time, which may lag due to data backlogs, service disruptions, etc. It's hard to beat the power of a live log tail when you need it.
    • otoolep 5y ago
      Yeah, agreed, if the logging agent (or whatever) isn't installed or running yet, it's going to be hard to get the logs into the centralized system. > That requires intermediate storage as logs are generated, which can fill up (disk or ram, it's all finite). That's also an argument for centralized logging. Don't store the logs anywhere near your production systems. We've all had the experience where a machine goes down because the disk is full.....due to log files.