11 ms·

The Syslog Hell

smetj 5y ago
> Parsing all of that mess is extremely “hacky”, with tons of regexes trying to account for all vendor quirks. Whilst that's possibly true, at least you/we have the possibility to do so.
- viraptor 5y ago
  What do you mean? You have this possibility with virtually every logs system. (On Linux anyway) Some of them just make it much easier.
  - jandrese 5y ago
    Try doing it with the Windows Event Viewer. SystemD's built in log thing at least has a text dump feature, but god help you if the database gets corrupted.
    - viraptor 5y ago
      That's why I said for Linux. Journald does not really have a database as such - it has index + sequential logs data. You can run the data through strings (unless you enabled compression) to get your data in emergency. You'd lose as much data from plaintext log corruption.
    - pengaru 5y ago
      > but god help you if the database gets corrupted What is this FUD nonsense you're attempting to spread here? systemd-journald assumes journals marked as ONLINE when opened for writing are potentially corrupted, and renames them rather than attempting to write to them, which produces messages like "journal corrupt, renaming". Those renamed journals still participate in journalctl queries, the data isn't lost. It's just a bit of wasted space in the interest of not risking actually corrupting an uncleanly shutdown journal by writing to it.
sam_lowry_ 5y ago
You have not seen the journalctl inferno yet.
- fullstop 5y ago
  I was forced to use systemd and journald for a while and I've seriously grown to like them, especially journald.
  - lathiat 5y ago
    Yep
  - hughrr 5y ago
    They’re really nice until everything on the node is completely broken. Then they are a massive obstruction to access and understanding what went wrong due to the opaqueness. It’s best to do some drills on a purposely broken system to gain some deep insight into recovery scenarios.
    - oblio 5y ago
      If everything is broken and you need to do recovery, don't you usually do that from another system anywhere? And mount the target system. I imagine the journald logs are just files at the end of the day and you can just read them with some tool?
    - fullstop 5y ago
      This can be tricky because journald compression, but is this really that much different than a corrupted compressed log file, like that which logrotate produces, in /var/log? If that is your concern, you can disable compression in the journald configuration so that the contents can be read with "strings" or similar tools.
    - theamk 5y ago
      I am curious what kind of problems have you seen? I am transitioning some of my systems to journald, and I am very interested in things that could go wrong. So far, I tried looking at logs from the dead system using “journalctl -D” - it seemed to work. And the way how the log files from each boot are always separate is pretty handy. Other than that, the only problems I have seen were having more to type and having to learn more commands. Am I in for a nasty surprise?
imglorp 5y ago
Any love for JSON logging when aiming for central aggregation? All kinds of benefits.
- tannhaeuser 5y ago
  Actually the syslog protocol has it's own structured format for log messages [1]. [1]: https://datatracker.ietf.org/doc/html/rfc5424 https://datatracker.ietf.org/doc/html/rfc5424
  - kevincox 5y ago
    Which as far as I can tell is implemented by next to nothing. Logging single-line JSON events will give you far more support for downstream tools.
- 0xbadcafebee 5y ago
  It wouldn't really change anything, it's just a data format. The problem is how to get people to both send and receive specific fields correctly. Lazy programmers/vendors would also still think they could throw together some crappy code and pretend it's generating JSON, but it would end up generating illegal JSON. Same for parsing. There are a lot of things you need in order to prevent the lazy from fucking up: - A version. A lazy programmer might either ignore or hard-code a version number, but at least you have a hint as to which standard you're trying to conform to, and can retain backwards compatibility. (The new RFC has a version, but the old one doesn't, preventing interoperability) - A format that's easy enough for programmers to understand, but difficult (or "feature-filled") enough that they won't attempt to implement it all themselves and will reach for real libraries. - Extensions. Vendors will always want to do something different than everyone else, so if you don't add the option of extensions, they will either fork the protocol and make breaking changes, or try to sneak changes into other parts of the protocol/format. - A standard reference implementation + tests. Make it easy for vendors to test their versions against another one, so the developers don't have to do busy-work like "read a standard" or "write tests". - Think about the future. Does your standard include a specific width integer? Does it preclude a specific network payload size? Will addressing change in the future? Can your data payload support arbitrary binary data? Can your standard change later and still be backwards compatible?
mrintegrity 5y ago
Started using Grafana Loki recently, so far it seems very good at consuming all the various log formats you might encounter and you can parse them into metrics etc as needed using regular expressions. Much nicer than dumping everything on a central syslog. My only "gripe" is the high learning curve compared to say elk stack.
- bogota 5y ago
  Does it really have a higher learning curve than running ELK? ELK is a nightmare to manage at any kind of scale in my experience. I guess I have been looking at Loki and is the issue in configuration or operation? If it’s harder to operate than ELK I’m not going to touch it though.
  - donio 5y ago
    I've only started to mess around with Loki but there is no way that it's more complex than a scaled up ELK stack.
  - mrintegrity 5y ago
    To be more clear, installing it and getting logs into the system is a piece of cake, getting useful information beyond the basics is more complicated and in my experience not as straight forward as elk stack. Kind of the exact opposite of elk in fact
latch 5y ago
A bit off topic, but we've been using Vector (1) for log ingestion, and I really like it. It's fast, low resources, actively developed and flexible. (1) https://vector.dev/ https://vector.dev/
- FridgeSeal 5y ago
  Seconding Vector here. Config was also way less painful than traipsing through whatever hellscape the FluentBit/FluentD configs are.
- e12e 5y ago
  Ingestion to where? I'm considering loki, but haven't quite ran enough test runs yet..
  - latch 5y ago
    A self-hosted ElasticSearch and Prometheus.
coldacid 5y ago
Don't get me started on Windows Event Log. Sure it might be better structured than syslog on paper, but it's even more of a mess. Not to mention that it's really hundreds of logs that just share a similar format and same UI.
- corty 5y ago
  And mostly the structure isn't really used at all, it's just plain text for a lot of log messages.
  - WorldMaker 5y ago
    Plain text that you cannot even access without a complicated COM control dance in many cases. It's an interesting design for sure. I have seen and hope to never see again worst cases such as the COM control for all of .NET Framework event log messages (everything from .NET system messages to just the mostly plain text storage from applications written in .NET) accidentally badly unregistered leaving all of the event log messages unreadable.
    - jjkaczor 5y ago
      "...applications written in .NET) accidentally badly unregistered leaving all of the event log messages unreadable..." Don't even get me started on this... Microsoft is actually bad for this with even their own .NET-based enterprise applications. As well - it also makes gathering logs from production servers, then performing analysis on a different machine difficult, as that machine will likely have none of the dependencies required. Text... text and more text, that is universal.
- jandrese 5y ago
  I would complain more, except I have never ever gotten a useful bit of information out of the Event Log. Not once. It's so laughably useless I'm surprised anybody even bothers anymore. Microsoft certainly doesn't. When applications on Windows fail they never think to generate an event, not even for something as simple as a "permission denied attempting to open file 'c:\blah'". Instead it's chock full of useless noise from daemons that activate every 2 seconds to poll something and then log that everything is still ok.
Stranger43 5y ago
And yet syslog works to the point where anything sold as an syslog replacement ends up adding complexity(along with features) rather then an simplification of the core problem. It's in general a trend for old unix tools to work better in reality then in theory something thats rare for more modern tools. Sure it's nice been able to use more modern query tools and have graphing libraries available but syslog grep and awk does get the job done and dont require a lot of resources to set up and maintain.
- viraptor 5y ago
  At the stage where you're looking at query tools and graphs, syslog stops being easy. How do you transfer logs? What's the naming convention for files? What's the lifecycle and what ensures it? How do multiple people access the logs? What happens with logs if network loses packets? That requires resources to design and keep running in practice. After dealing with a few systems for logs, I'd rather choose them for non-trivial setup now, than bare syslog and redo all of this from scratch.
  - touisteur 5y ago
    Ideally you stream logs and try not to rely on local storage. Rsyslog is quite the work beast and has a multitude of useful features (e.g. tcp+round-robin-multipath transfer mechanisms, disk-backed queues, rate limitation) and if you're feeling it, writing extensions is not that hard. Just reading all the docs is mind-boggling...
- nimbius 5y ago
  Anyone whos been sold Splunk as a drop-in syslog replacement for observability or visibility is painfully aware of this. Larger companies that use splunk as a central auditing tool are invariably left with a byzantine nightmare of dashboards and strings to figure out. people leave and roles change, and eventually youre running a nine year old server that can hardly tell you the time, let alone the state of nginx. just write a script, email the people regularly in charge with the CSV, and let the PHB make it look pretty.
  - 5y ago
alecco 5y ago
Being a maintainer of a secure syslog implementation around 2001, I was asked to join the IETF syslog group. There were a couple of syslog-ng guys, a Cisco guy, and I think a Microsoft guy. Plus a couple of randoms that didn't participate. The Cisco/Microsoft guys there looked like people whose sole job was to sit on standard comitees. They were not developers, at least not as a day job. I was trying to reach a compromise in a simple syslog standard so it would be easier to authenticate and analyze. And trying to make it good enough for non-*nix systems. Nobody else cared about this. It was one of the worst time wasters in my life. It was all politics. The syslog-ng guys were adamant with their proposal which was a very, very over-complicated idea based on another standard (BEEP). And I strongly suspect the Cisco/Microsoft guys were intentionally trying to make the group not work in subtle ways. After months, I just left. They eventually published RFC 3195. And it's barely used, of course. It seems Cisco's implementation still uses DIGEST-MD5 for authentication.
- 0xbadcafebee 5y ago
  Sorry you had to go through that, but standards bodies are literally political bodies. That's why the commercial representatives are not devs. They are there to advocate for their own constituents, not help others. The Cisco/Microsoft guys just wanted to make sure nobody would break their products, and that their products would work with other things. That generally means changing as little as possible, which can definitely look like "not working".
nobleach 5y ago
When I started playing with Linux back around 1996, I started learning the real value of having logs. It took me into my first career in IT. The Unix/Linux value behind logging everything and quickly getting to the bottom of any problem, was something that was missing from my MS Windows experience. Yet the Event Viewer existed. No, it was rarely all that useful. Somehow lines in a text file... rotated at a predetermined interval... is just so simple yet, completely effective. Through the years I've tried the other tools that are supposed to supplant it yet, I still think it was the best. I'm getting used to the SystemD/JournalD way but, I really did like having a directory full of text files (and gzipped friends from previous days)
jascii 5y ago
Syslog "Hell"? Ever looked at SNMP?! When a "standard" sticks around this long and needs to support so many legacy devices things can get a bit messy. At least syslog is human readable, while things may not be as machine parsable as you'd like, the info you need is usually only a few greps away.
- jandrese 5y ago
  SNMP is at least a real standard, well several real standards, that are sometimes even followed. The big problem with SNMP is that the MIBs have to be handled out of band. If there was some part of the standard where you could query the device to get its MIB in some standard format it would be so so much better. The daemon could be small because it wouldn't have to ship with hundreds of megabytes of data for devices built over the past 40 years. You wouldn't have to go on a hunt to track down where the vendor hid the MIBs for oddball and obsolete equipment, often times only available with a support contract on a website that was decommissioned years ago. Alternatively it could have a query type that gives you a description of every field, so when you walk the tree you get all of the data that you would otherwise need the MIB for.
miga 5y ago
Did you try Pcapng specification? I believe it is used by new syslog, and this is the packet format for logging both network interfaces, syslog files, and many more things together in a single format. https://wiki.wireshark.org/SampleCaptures https://wiki.wireshark.org/SampleCaptures https://wiki.wireshark.org/Development/PcapNg https://wiki.wireshark.org/Development/PcapNg
- touisteur 5y ago
  I second this. Pcapng is just a very simple, extensible, file format. I actually use it for anything that needs timestamping and a block-sequence format, for post-processing.
floren 5y ago
I feel the pain. Everybody decides their appliance will emit "syslog", but they don't bother to look at either RFC (not even the very lax RFC3164) and just emit "log messages preceded by some sort of timestamp", as the article calls out: > But what makes things hell is the fact that too many vendors decided not to care about what is in the RFCs, they decided that “hey, putting a year there is just fine” even though the RFC says “no”, that they don’t really need to set a host in the header, and that they didn’t really need to implement anything new after their initial legacy stuff was created. It sounds like the author and I are doing similar work, so he knows my pain: if you make a product which can parse syslog, and somebody selects your product for parsing syslog, and they they feed it non-syslog logs from Company Y's product... it's now your problem, instead of Company Y's, even though you're perfectly capable of parsing syslog! Luckily, regular expressions and beer eventually get most things sorted out. :)
dale_glass 5y ago
And that's why journald is such a cool thing. * Want to parse stuff? journalctl -o json * Lots of stuff going on, need more precise timestamps? -o short-precise * Want metadata, like the pid? It's in there. * Want to know where to continue parsing? It supports cursors. * Want to save disk space? It uncompresses logs transparently and can trim logs to whatever size you want.
- InvertedRhodium 5y ago
  Want to filter logs at collection time? Well, too bad - your use case is impure and you should just fix everything immediately, even that over which you have no direct control. https://github.com/systemd/systemd/issues/6432 https://github.com/systemd/systemd/issues/6432
  - Poiesis 5y ago
    I realize that the linked thread explains that decreasing the log level wouldn't work because everything else would be affected—but couldn't someone combine decreasing the log level with using a different journal namespace?
  - dale_glass 5y ago
    I actually mostly agree with Poettering here that it's kind of a hack and that the ideal is to collect everything and filter afterwards. You can't get back stuff you exclude by mistake. Programs should be logging because there's some value in the information being sent to the log. If it annoys everyone and serves no purpose, the program needs fixing.
    - eximius 5y ago
      The programs needing fixing and supporting filtering are not exclusive options.
    - InvertedRhodium 5y ago
      I agree that they _should_ be fixed - but life isn't perfect, and we don't always have a way to fix the issue ourselves. > You can't get back stuff you exclude by mistake. Ok. That's fine - it's my system, my data, and my ass on the line if we lose data by mistake. I don't know see why a developer that knows nothing about my problem domain and the constraints under which I'm operating gets to dictate to me that my choice is wrong.
nwmcsween 5y ago
I've thought about this a bit and ideally one could either use a compressor with a predefined dict populated with normal daemon syslog output and output based on bad compression ratio of syslog messages or use minwise hashing and have a learning mode to populate a 'good' table and similar to the compression close jaccard similarities get discarded.
marviio 5y ago
Secure collection and transfer of logs, decorated with metadata (timestamp(s), logging host, mac address, pid etc.) is solved many times. Just pick a tool. The hard part is the message field's content and format. It basically boils down to actions of thousands of individual developers. They will never agree on a format and logging style.

> Parsing all of that mess is extremely “hacky”, with tons of regexes trying to account for all vendor quirks. Whilst that's possibly true, at least you/we have the possibility to do so.

viraptor 5y ago
What do you mean? You have this possibility with virtually every logs system. (On Linux anyway) Some of them just make it much easier.
- jandrese 5y ago
  Try doing it with the Windows Event Viewer. SystemD's built in log thing at least has a text dump feature, but god help you if the database gets corrupted.
  - viraptor 5y ago
    That's why I said for Linux. Journald does not really have a database as such - it has index + sequential logs data. You can run the data through strings (unless you enabled compression) to get your data in emergency. You'd lose as much data from plaintext log corruption.
  - pengaru 5y ago
    > but god help you if the database gets corrupted What is this FUD nonsense you're attempting to spread here? systemd-journald assumes journals marked as ONLINE when opened for writing are potentially corrupted, and renames them rather than attempting to write to them, which produces messages like "journal corrupt, renaming". Those renamed journals still participate in journalctl queries, the data isn't lost. It's just a bit of wasted space in the interest of not risking actually corrupting an uncleanly shutdown journal by writing to it.

sam_lowry_ 5y ago

You have not seen the journalctl inferno yet.

fullstop 5y ago
I was forced to use systemd and journald for a while and I've seriously grown to like them, especially journald.
- lathiat 5y ago
  Yep
- hughrr 5y ago
  They’re really nice until everything on the node is completely broken. Then they are a massive obstruction to access and understanding what went wrong due to the opaqueness. It’s best to do some drills on a purposely broken system to gain some deep insight into recovery scenarios.
  - oblio 5y ago
    If everything is broken and you need to do recovery, don't you usually do that from another system anywhere? And mount the target system. I imagine the journald logs are just files at the end of the day and you can just read them with some tool?
  - fullstop 5y ago
    This can be tricky because journald compression, but is this really that much different than a corrupted compressed log file, like that which logrotate produces, in /var/log? If that is your concern, you can disable compression in the journald configuration so that the contents can be read with "strings" or similar tools.
  - theamk 5y ago
    I am curious what kind of problems have you seen? I am transitioning some of my systems to journald, and I am very interested in things that could go wrong. So far, I tried looking at logs from the dead system using “journalctl -D” - it seemed to work. And the way how the log files from each boot are always separate is pretty handy. Other than that, the only problems I have seen were having more to type and having to learn more commands. Am I in for a nasty surprise?

imglorp 5y ago

Any love for JSON logging when aiming for central aggregation? All kinds of benefits.

tannhaeuser 5y ago
Actually the syslog protocol has it's own structured format for log messages [1]. [1]: https://datatracker.ietf.org/doc/html/rfc5424 https://datatracker.ietf.org/doc/html/rfc5424
- kevincox 5y ago
  Which as far as I can tell is implemented by next to nothing. Logging single-line JSON events will give you far more support for downstream tools.
0xbadcafebee 5y ago
It wouldn't really change anything, it's just a data format. The problem is how to get people to both send and receive specific fields correctly. Lazy programmers/vendors would also still think they could throw together some crappy code and pretend it's generating JSON, but it would end up generating illegal JSON. Same for parsing. There are a lot of things you need in order to prevent the lazy from fucking up: - A version. A lazy programmer might either ignore or hard-code a version number, but at least you have a hint as to which standard you're trying to conform to, and can retain backwards compatibility. (The new RFC has a version, but the old one doesn't, preventing interoperability) - A format that's easy enough for programmers to understand, but difficult (or "feature-filled") enough that they won't attempt to implement it all themselves and will reach for real libraries. - Extensions. Vendors will always want to do something different than everyone else, so if you don't add the option of extensions, they will either fork the protocol and make breaking changes, or try to sneak changes into other parts of the protocol/format. - A standard reference implementation + tests. Make it easy for vendors to test their versions against another one, so the developers don't have to do busy-work like "read a standard" or "write tests". - Think about the future. Does your standard include a specific width integer? Does it preclude a specific network payload size? Will addressing change in the future? Can your data payload support arbitrary binary data? Can your standard change later and still be backwards compatible?

mrintegrity 5y ago

Started using Grafana Loki recently, so far it seems very good at consuming all the various log formats you might encounter and you can parse them into metrics etc as needed using regular expressions. Much nicer than dumping everything on a central syslog. My only "gripe" is the high learning curve compared to say elk stack.

bogota 5y ago
Does it really have a higher learning curve than running ELK? ELK is a nightmare to manage at any kind of scale in my experience. I guess I have been looking at Loki and is the issue in configuration or operation? If it’s harder to operate than ELK I’m not going to touch it though.
- donio 5y ago
  I've only started to mess around with Loki but there is no way that it's more complex than a scaled up ELK stack.
- mrintegrity 5y ago
  To be more clear, installing it and getting logs into the system is a piece of cake, getting useful information beyond the basics is more complicated and in my experience not as straight forward as elk stack. Kind of the exact opposite of elk in fact

latch 5y ago

A bit off topic, but we've been using Vector (1) for log ingestion, and I really like it. It's fast, low resources, actively developed and flexible. (1) https://vector.dev/ https://vector.dev/

FridgeSeal 5y ago
Seconding Vector here. Config was also way less painful than traipsing through whatever hellscape the FluentBit/FluentD configs are.
e12e 5y ago
Ingestion to where? I'm considering loki, but haven't quite ran enough test runs yet..
- latch 5y ago
  A self-hosted ElasticSearch and Prometheus.

coldacid 5y ago

Don't get me started on Windows Event Log. Sure it might be better structured than syslog on paper, but it's even more of a mess. Not to mention that it's really hundreds of logs that just share a similar format and same UI.

corty 5y ago
And mostly the structure isn't really used at all, it's just plain text for a lot of log messages.
- WorldMaker 5y ago
  Plain text that you cannot even access without a complicated COM control dance in many cases. It's an interesting design for sure. I have seen and hope to never see again worst cases such as the COM control for all of .NET Framework event log messages (everything from .NET system messages to just the mostly plain text storage from applications written in .NET) accidentally badly unregistered leaving all of the event log messages unreadable.
  - jjkaczor 5y ago
    "...applications written in .NET) accidentally badly unregistered leaving all of the event log messages unreadable..." Don't even get me started on this... Microsoft is actually bad for this with even their own .NET-based enterprise applications. As well - it also makes gathering logs from production servers, then performing analysis on a different machine difficult, as that machine will likely have none of the dependencies required. Text... text and more text, that is universal.
jandrese 5y ago
I would complain more, except I have never ever gotten a useful bit of information out of the Event Log. Not once. It's so laughably useless I'm surprised anybody even bothers anymore. Microsoft certainly doesn't. When applications on Windows fail they never think to generate an event, not even for something as simple as a "permission denied attempting to open file 'c:\blah'". Instead it's chock full of useless noise from daemons that activate every 2 seconds to poll something and then log that everything is still ok.

Stranger43 5y ago

And yet syslog works to the point where anything sold as an syslog replacement ends up adding complexity(along with features) rather then an simplification of the core problem. It's in general a trend for old unix tools to work better in reality then in theory something thats rare for more modern tools. Sure it's nice been able to use more modern query tools and have graphing libraries available but syslog grep and awk does get the job done and dont require a lot of resources to set up and maintain.

viraptor 5y ago
At the stage where you're looking at query tools and graphs, syslog stops being easy. How do you transfer logs? What's the naming convention for files? What's the lifecycle and what ensures it? How do multiple people access the logs? What happens with logs if network loses packets? That requires resources to design and keep running in practice. After dealing with a few systems for logs, I'd rather choose them for non-trivial setup now, than bare syslog and redo all of this from scratch.
- touisteur 5y ago
  Ideally you stream logs and try not to rely on local storage. Rsyslog is quite the work beast and has a multitude of useful features (e.g. tcp+round-robin-multipath transfer mechanisms, disk-backed queues, rate limitation) and if you're feeling it, writing extensions is not that hard. Just reading all the docs is mind-boggling...
nimbius 5y ago
Anyone whos been sold Splunk as a drop-in syslog replacement for observability or visibility is painfully aware of this. Larger companies that use splunk as a central auditing tool are invariably left with a byzantine nightmare of dashboards and strings to figure out. people leave and roles change, and eventually youre running a nine year old server that can hardly tell you the time, let alone the state of nginx. just write a script, email the people regularly in charge with the CSV, and let the PHB make it look pretty.
- 5y ago

alecco 5y ago

Being a maintainer of a secure syslog implementation around 2001, I was asked to join the IETF syslog group. There were a couple of syslog-ng guys, a Cisco guy, and I think a Microsoft guy. Plus a couple of randoms that didn't participate. The Cisco/Microsoft guys there looked like people whose sole job was to sit on standard comitees. They were not developers, at least not as a day job. I was trying to reach a compromise in a simple syslog standard so it would be easier to authenticate and analyze. And trying to make it good enough for non-*nix systems. Nobody else cared about this. It was one of the worst time wasters in my life. It was all politics. The syslog-ng guys were adamant with their proposal which was a very, very over-complicated idea based on another standard (BEEP). And I strongly suspect the Cisco/Microsoft guys were intentionally trying to make the group not work in subtle ways. After months, I just left. They eventually published RFC 3195. And it's barely used, of course. It seems Cisco's implementation still uses DIGEST-MD5 for authentication.

0xbadcafebee 5y ago
Sorry you had to go through that, but standards bodies are literally political bodies. That's why the commercial representatives are not devs. They are there to advocate for their own constituents, not help others. The Cisco/Microsoft guys just wanted to make sure nobody would break their products, and that their products would work with other things. That generally means changing as little as possible, which can definitely look like "not working".

nobleach 5y ago

When I started playing with Linux back around 1996, I started learning the real value of having logs. It took me into my first career in IT. The Unix/Linux value behind logging everything and quickly getting to the bottom of any problem, was something that was missing from my MS Windows experience. Yet the Event Viewer existed. No, it was rarely all that useful. Somehow lines in a text file... rotated at a predetermined interval... is just so simple yet, completely effective. Through the years I've tried the other tools that are supposed to supplant it yet, I still think it was the best. I'm getting used to the SystemD/JournalD way but, I really did like having a directory full of text files (and gzipped friends from previous days)

jascii 5y ago

Syslog "Hell"? Ever looked at SNMP?! When a "standard" sticks around this long and needs to support so many legacy devices things can get a bit messy. At least syslog is human readable, while things may not be as machine parsable as you'd like, the info you need is usually only a few greps away.

jandrese 5y ago
SNMP is at least a real standard, well several real standards, that are sometimes even followed. The big problem with SNMP is that the MIBs have to be handled out of band. If there was some part of the standard where you could query the device to get its MIB in some standard format it would be so so much better. The daemon could be small because it wouldn't have to ship with hundreds of megabytes of data for devices built over the past 40 years. You wouldn't have to go on a hunt to track down where the vendor hid the MIBs for oddball and obsolete equipment, often times only available with a support contract on a website that was decommissioned years ago. Alternatively it could have a query type that gives you a description of every field, so when you walk the tree you get all of the data that you would otherwise need the MIB for.

miga 5y ago

Did you try Pcapng specification? I believe it is used by new syslog, and this is the packet format for logging both network interfaces, syslog files, and many more things together in a single format. https://wiki.wireshark.org/SampleCaptures https://wiki.wireshark.org/SampleCaptures https://wiki.wireshark.org/Development/PcapNg https://wiki.wireshark.org/Development/PcapNg

touisteur 5y ago
I second this. Pcapng is just a very simple, extensible, file format. I actually use it for anything that needs timestamping and a block-sequence format, for post-processing.

floren 5y ago

I feel the pain. Everybody decides their appliance will emit "syslog", but they don't bother to look at either RFC (not even the very lax RFC3164) and just emit "log messages preceded by some sort of timestamp", as the article calls out: > But what makes things hell is the fact that too many vendors decided not to care about what is in the RFCs, they decided that “hey, putting a year there is just fine” even though the RFC says “no”, that they don’t really need to set a host in the header, and that they didn’t really need to implement anything new after their initial legacy stuff was created. It sounds like the author and I are doing similar work, so he knows my pain: if you make a product which can parse syslog, and somebody selects your product for parsing syslog, and they they feed it non-syslog logs from Company Y's product... it's now your problem, instead of Company Y's, even though you're perfectly capable of parsing syslog! Luckily, regular expressions and beer eventually get most things sorted out. :)

dale_glass 5y ago

And that's why journald is such a cool thing. * Want to parse stuff? journalctl -o json * Lots of stuff going on, need more precise timestamps? -o short-precise * Want metadata, like the pid? It's in there. * Want to know where to continue parsing? It supports cursors. * Want to save disk space? It uncompresses logs transparently and can trim logs to whatever size you want.

InvertedRhodium 5y ago
Want to filter logs at collection time? Well, too bad - your use case is impure and you should just fix everything immediately, even that over which you have no direct control. https://github.com/systemd/systemd/issues/6432 https://github.com/systemd/systemd/issues/6432
- Poiesis 5y ago
  I realize that the linked thread explains that decreasing the log level wouldn't work because everything else would be affected—but couldn't someone combine decreasing the log level with using a different journal namespace?
- dale_glass 5y ago
  I actually mostly agree with Poettering here that it's kind of a hack and that the ideal is to collect everything and filter afterwards. You can't get back stuff you exclude by mistake. Programs should be logging because there's some value in the information being sent to the log. If it annoys everyone and serves no purpose, the program needs fixing.
  - eximius 5y ago
    The programs needing fixing and supporting filtering are not exclusive options.
  - InvertedRhodium 5y ago
    I agree that they _should_ be fixed - but life isn't perfect, and we don't always have a way to fix the issue ourselves. > You can't get back stuff you exclude by mistake. Ok. That's fine - it's my system, my data, and my ass on the line if we lose data by mistake. I don't know see why a developer that knows nothing about my problem domain and the constraints under which I'm operating gets to dictate to me that my choice is wrong.

nwmcsween 5y ago

I've thought about this a bit and ideally one could either use a compressor with a predefined dict populated with normal daemon syslog output and output based on bad compression ratio of syslog messages or use minwise hashing and have a learning mode to populate a 'good' table and similar to the compression close jaccard similarities get discarded.

marviio 5y ago

Secure collection and transfer of logs, decorated with metadata (timestamp(s), logging host, mac address, pid etc.) is solved many times. Just pick a tool. The hard part is the message field's content and format. It basically boils down to actions of thousands of individual developers. They will never agree on a format and logging style.