3 ms·
> Dedupe/compression works really well on syslog servers where I've seen 6:1 savings. Don’t you compress these directly? I normally see at least twice that for
by acdha 2y ago
> Dedupe/compression works really well on syslog servers where I've seen 6:1 savings.
Don’t you compress these directly? I normally see at least twice that for logs doing it at the process level.
- UltraSane 2y agoWhat software?
- SteveNuts 2y agoLogrotate
- acdha 2y agoLog rotate, cron, or simply having something like Varnish or Apache log to a pipe which is something like bzip2 or zstd. The main question is whether you want to easily access the current stream - e.g. I had uncompressed logs being forwarded to CloudWatch so I had daemons logging to timestamped files with a post-rotate compression command which would run after the last write.
- UltraSane 2y agoThat is one wrinkle of using storage based dedupe/compression is you need to avoid doing compression on the client to avoid compressing already compressed data. When a company I worked at first got their Pure array they were using windows file compression heavily and had to disable it as the storage array was now doing it automatically.
- acdha 2y agoDefinitely. We love building abstraction layers but at some point you really need to make decisions across the entire stack.
- chasil 2y agoLogrotate is the rhel utility, likely present in Fedora, that is easily adapted for custom log handling. I still have rhel5 and I use it there. CentOS made it famous. I don't know if it has a foothold in the Debian family.
- E39M5S62 2y agologrotate is used on Debian and plenty of other distros. It seems pretty widely used, though maybe not as much so now that things log through systemd.
- pezezin 2y agoYes, that ratio is very small. I built a very simple, custom syslog solution, a syslog-ng server writing directly to a TimescaleDB hypertable (https://www.timescale.com/ https://www.timescale.com/) that is then presented as a Grafana dashboard, and I am getting a 30x compression ratio.
- pdimitar 2y agoWould love to see your solution -- if it's open source.
- deleted 2y ago[deleted]
- pezezin 2y agoSure, no problem. But after you asked me, I realized that the system was not properly documented anywhere, I didn't even have a repo with the configuration files, what an embarrassment :( I just created the repo and uploaded the documentation, please give me some more time to write the documentation: https://github.com/josefrcm/simple-syslog-service https://github.com/josefrcm/simple-syslog-service