7 ms·

And yet syslog works to the point where anything sold as an syslog replacement ends up adding complexity(along with features) rather then an simplification of t

And yet syslog works to the point where anything sold as an syslog replacement ends up adding complexity(along with features) rather then an simplification of the core problem. It's in general a trend for old unix tools to work better in reality then in theory something thats rare for more modern tools. Sure it's nice been able to use more modern query tools and have graphing libraries available but syslog grep and awk does get the job done and dont require a lot of resources to set up and maintain.

viraptor 5y ago
At the stage where you're looking at query tools and graphs, syslog stops being easy. How do you transfer logs? What's the naming convention for files? What's the lifecycle and what ensures it? How do multiple people access the logs? What happens with logs if network loses packets? That requires resources to design and keep running in practice. After dealing with a few systems for logs, I'd rather choose them for non-trivial setup now, than bare syslog and redo all of this from scratch.
- touisteur 5y ago
  Ideally you stream logs and try not to rely on local storage. Rsyslog is quite the work beast and has a multitude of useful features (e.g. tcp+round-robin-multipath transfer mechanisms, disk-backed queues, rate limitation) and if you're feeling it, writing extensions is not that hard. Just reading all the docs is mind-boggling...
nimbius 5y ago
Anyone whos been sold Splunk as a drop-in syslog replacement for observability or visibility is painfully aware of this. Larger companies that use splunk as a central auditing tool are invariably left with a byzantine nightmare of dashboards and strings to figure out. people leave and roles change, and eventually youre running a nine year old server that can hardly tell you the time, let alone the state of nginx. just write a script, email the people regularly in charge with the CSV, and let the PHB make it look pretty.
- znpy 5y ago
  > and eventually youre running a nine year old server that can hardly tell you the time, let alone the state of nginx. thank you for the good laugh :)
cb321 5y ago
I think this is more that "more features" = "more marketable" which impacts far more than just "unix tools" or even just software. Also, simplifiation does not never happen. E.g., I have a near trivial impl (under 300 lines of Nim) at https://github.com/c-blake/kslog https://github.com/c-blake/kslog { yeah, it may not be of very wide appeal..See the first point. :-) } I feel a better factoring/separation of concerns is to disentangle file/data distribution from getting in-socket data somewhere persistent.
dale_glass 5y ago
Yeah, no. Maybe that worked in the 80s, back when hardware was weak, and an admin could keep an eye on everything by hand. I dealt with parsing log files more recently than that, and it's a never-ending list of annoying bullshit to deal with: * Some stuff logs with syslog and some doesn't. * Formats vary, oh the fun of implementing the different variations. * You have to parse text dates into unix, when whatever wrote the file converted unix to text. It probably lacks milliseconds, which is all kinds of fun in a modern setting where there can easily be a hundred things happening during any second. * Various edge cases. Where exactly does every field in the log file end? Can there be a newline? (yes, guaranteed). Can there be random binary junk (yup, sometimes). * How do you keep track of where you stopped parsing? How do you deal with that the old log might have been removed and a new one with the same name now appeared? * Dealing with compression, log rotation, race conditions. It's simple on the surface. Actually writing a program that deals with all that stuff is bloody annoying, because none of it actually gets what you want done. You typically want to detect important events happening, or graphing something. Instead, 95% of the time goes on mind-numbing minutia dealing with parsing because the system was built for an admin using `tail` and `grep` in the 80s. It wasn't planned for a modern admin maintaining a few dozen computers each of which log multiple megabytes of stuff every hour. Never understood people who complain about journald, because this stuff was a pain in my butt a good decade before journald existed. I certainly don't have any fond memories from dealing with it.
- touisteur 5y ago
  I'm guessing i would have preferred to standardise something in syslog-ng and rsyslog, like a database-backed log-file format (sqlite?) instead of yet another piece of the yuge systemd hydra...
  - dale_glass 5y ago
    sqlite doesn't sound like a very good choice to me. journald's log format is nearly \0 separated fields, while sqlite is a good deal more complex than that. Also journald allows for append-only logs, which I don't think sqlite supports. And personally I see the "hydra" as a benefit -- everything integrates well with everything else, because it's all designed to go well together.

viraptor 5y ago

At the stage where you're looking at query tools and graphs, syslog stops being easy. How do you transfer logs? What's the naming convention for files? What's the lifecycle and what ensures it? How do multiple people access the logs? What happens with logs if network loses packets? That requires resources to design and keep running in practice. After dealing with a few systems for logs, I'd rather choose them for non-trivial setup now, than bare syslog and redo all of this from scratch.

touisteur 5y ago
Ideally you stream logs and try not to rely on local storage. Rsyslog is quite the work beast and has a multitude of useful features (e.g. tcp+round-robin-multipath transfer mechanisms, disk-backed queues, rate limitation) and if you're feeling it, writing extensions is not that hard. Just reading all the docs is mind-boggling...

nimbius 5y ago

Anyone whos been sold Splunk as a drop-in syslog replacement for observability or visibility is painfully aware of this. Larger companies that use splunk as a central auditing tool are invariably left with a byzantine nightmare of dashboards and strings to figure out. people leave and roles change, and eventually youre running a nine year old server that can hardly tell you the time, let alone the state of nginx. just write a script, email the people regularly in charge with the CSV, and let the PHB make it look pretty.

znpy 5y ago
> and eventually youre running a nine year old server that can hardly tell you the time, let alone the state of nginx. thank you for the good laugh :)

cb321 5y ago

I think this is more that "more features" = "more marketable" which impacts far more than just "unix tools" or even just software. Also, simplifiation does not never happen. E.g., I have a near trivial impl (under 300 lines of Nim) at https://github.com/c-blake/kslog https://github.com/c-blake/kslog { yeah, it may not be of very wide appeal..See the first point. :-) } I feel a better factoring/separation of concerns is to disentangle file/data distribution from getting in-socket data somewhere persistent.

dale_glass 5y ago

Yeah, no. Maybe that worked in the 80s, back when hardware was weak, and an admin could keep an eye on everything by hand. I dealt with parsing log files more recently than that, and it's a never-ending list of annoying bullshit to deal with: * Some stuff logs with syslog and some doesn't. * Formats vary, oh the fun of implementing the different variations. * You have to parse text dates into unix, when whatever wrote the file converted unix to text. It probably lacks milliseconds, which is all kinds of fun in a modern setting where there can easily be a hundred things happening during any second. * Various edge cases. Where exactly does every field in the log file end? Can there be a newline? (yes, guaranteed). Can there be random binary junk (yup, sometimes). * How do you keep track of where you stopped parsing? How do you deal with that the old log might have been removed and a new one with the same name now appeared? * Dealing with compression, log rotation, race conditions. It's simple on the surface. Actually writing a program that deals with all that stuff is bloody annoying, because none of it actually gets what you want done. You typically want to detect important events happening, or graphing something. Instead, 95% of the time goes on mind-numbing minutia dealing with parsing because the system was built for an admin using `tail` and `grep` in the 80s. It wasn't planned for a modern admin maintaining a few dozen computers each of which log multiple megabytes of stuff every hour. Never understood people who complain about journald, because this stuff was a pain in my butt a good decade before journald existed. I certainly don't have any fond memories from dealing with it.

touisteur 5y ago
I'm guessing i would have preferred to standardise something in syslog-ng and rsyslog, like a database-backed log-file format (sqlite?) instead of yet another piece of the yuge systemd hydra...
- dale_glass 5y ago
  sqlite doesn't sound like a very good choice to me. journald's log format is nearly \0 separated fields, while sqlite is a good deal more complex than that. Also journald allows for append-only logs, which I don't think sqlite supports. And personally I see the "hydra" as a benefit -- everything integrates well with everything else, because it's all designed to go well together.