4 ms·

Do not invent your own formats. There is an RFC for structured logging [1] Also, journald does structured logging, plus indexing and searching like a simple d

Do not invent your own formats. There is an RFC for structured logging [1] Also, journald does structured logging, plus indexing and searching like a simple database and it's designed for your use case. It can receive the logs and forward them using a connector for ElasticSearch [2] [1] https://tools.ietf.org/html/rfc5424 https://tools.ietf.org/html/rfc5424 [2] https://www.loggly.com/blog/why-journald/ https://www.loggly.com/blog/why-journald/

paulddraper 10y ago
RFC5424 Syslog defines some the common stuff all logs should have: hostname, app name, datetime, message, priority. Many logging tools (Splunk, Loggly, etc.) consume syslog. Syslog works over UDP and TCP. Java Logback has a syslog appender. Linux systems have rsyslog for forwarding, writing, etc. You'll wind up including your own custom structured data of course (IP address, response time, user id etc.), but it's nice use a recognized standard for the standard parts. (Kind of like, say, using status code, cache-control, and content-type in HTTP.)
- eeZah7Ux 10y ago
  The RFC defines key-value structured logging. https://tools.ietf.org/html/rfc5424#section-6.3.5 https://tools.ietf.org/html/rfc5424#section-6.3.5
nailer 10y ago
I got excited reading your second link (https://www.loggly.com/blog/why-journald/ https://www.loggly.com/blog/why-journald/ ), but from it: > Sadly, journald does not come with a usable remote logging solution. The program systemd-journal-remote is more of a proof-of-concept than an actually useful tool, lacking good authentication among other things. Third-party packages such as journald-forwarder send journal logs to Loggly directly but are very new and experimental. As such, remote logging still has to go through existing syslog implementations and the syslog protocol and therefore cannot make use of many of the additional features of journald, in particular the structured data. Anyone on HN cracked structured remote centralised logging with journald? Unstructuring everything for syslog, dealing with pretend stuff like uucp, fax and user3, and restructuring again seems like a kludge. Also: weird loggly don't sieze the opportunity and make something that works with their service without the limitations they're writing about.
- otterley 10y ago
  It's not as difficult as it's made out to be. We run a systemd service that does something like this: journalctl -f -o json | send-logs remote-server
  - eeZah7Ux 10y ago
    And there are also tools like https://github.com/aiven/journalpump https://github.com/aiven/journalpump
  - nailer 10y ago
    What is send-logs and what protocol does it use?
rjbwork 10y ago
Speaking of Loggly, I highly recommend them as a log analysis sink.

paulddraper 10y ago

RFC5424 Syslog defines some the common stuff all logs should have: hostname, app name, datetime, message, priority. Many logging tools (Splunk, Loggly, etc.) consume syslog. Syslog works over UDP and TCP. Java Logback has a syslog appender. Linux systems have rsyslog for forwarding, writing, etc. You'll wind up including your own custom structured data of course (IP address, response time, user id etc.), but it's nice use a recognized standard for the standard parts. (Kind of like, say, using status code, cache-control, and content-type in HTTP.)

eeZah7Ux 10y ago
The RFC defines key-value structured logging. https://tools.ietf.org/html/rfc5424#section-6.3.5 https://tools.ietf.org/html/rfc5424#section-6.3.5

nailer 10y ago

I got excited reading your second link (https://www.loggly.com/blog/why-journald/ https://www.loggly.com/blog/why-journald/ ), but from it: > Sadly, journald does not come with a usable remote logging solution. The program systemd-journal-remote is more of a proof-of-concept than an actually useful tool, lacking good authentication among other things. Third-party packages such as journald-forwarder send journal logs to Loggly directly but are very new and experimental. As such, remote logging still has to go through existing syslog implementations and the syslog protocol and therefore cannot make use of many of the additional features of journald, in particular the structured data. Anyone on HN cracked structured remote centralised logging with journald? Unstructuring everything for syslog, dealing with pretend stuff like uucp, fax and user3, and restructuring again seems like a kludge. Also: weird loggly don't sieze the opportunity and make something that works with their service without the limitations they're writing about.

otterley 10y ago
It's not as difficult as it's made out to be. We run a systemd service that does something like this: journalctl -f -o json | send-logs remote-server
- eeZah7Ux 10y ago
  And there are also tools like https://github.com/aiven/journalpump https://github.com/aiven/journalpump
- nailer 10y ago
  What is send-logs and what protocol does it use?

rjbwork 10y ago

Speaking of Loggly, I highly recommend them as a log analysis sink.