3 ms·
> Just writing stuff to a file was fine. Haha, I wish. Then that file becomes too large. So you want to truncate the start. However, removing bytes from the st
by ATsch 5y ago
> Just writing stuff to a file was fine.
Haha, I wish. Then that file becomes too large.
So you want to truncate the start. However, removing bytes from the start of an open file is a bad idea.
So you open a new log file every day instead. You really want to compress the old log files too, to save space.
So now you can no longer just grep the logs as before. You're now also duplicating code and configuration for every daemon.
So you write a daemon called `logrotate` that does this centrally instead. Into which you then copy the log paths configured for every program on your system and how long the logs should get kept. However programs don't much like it when you move the log files out from under them.
So you patch every program respond to a signal by checking whether it's log files have been moved. Which then breaks for multi-process daemons.
You come up with a whole buffet of ugly hacks to make that work most of the time.
Then you install a program which doesn't come with a logrotate config.
Then you decide you'd really want a more structured way to analyze logs than writing fragile regexes.
Then you want to run something as a non-root user.
Then you have an ephermeral batch job.
Then you want to ship off all of your logs to another location and really don't want to duplicate the log paths yet again.
Then you want to see what happened just before the server crashed last boot.
Then you run two of a daemon and give them different log paths.
Then you want to look at your kernel and application logs together.
Then you want to delete just last month's logs to free up some space.
Also wouldn't it be handy if there was some kind of index to make things faster?
...
- Arch-TK 5y agoAll of this seems solved by svlogd honestly.
- ATsch 5y agoI have never used svlogd but at it looks like it does read logs from it's stdin, which is a solid step forward as it avoids much of the pitfalls caused by applications writing directly into log files. Collection is just the first part of a log system though of course.
- magicalhippo 5y ago> However, removing bytes from the start of an open file is a bad idea. A digression, but I keep wondering why filesystems never optimized this scenario. Being able to trim the start of the file is really useful in certain cases, and without support options are pretty bad. It also seems fairly easy to support at the cost of a few bytes in the file entry (a start-of-data index into first sector), and resizing a file by seeking past end is already a thing. I must be missing something...