5 ms·

Honestly, I'd say that logging should be text. Yes, SQLite DBs are hard to corrupt, but it can happen, especially in the sort of catastrophic failure you would

Honestly, I'd say that logging should be text. Yes, SQLite DBs are hard to corrupt, but it can happen, especially in the sort of catastrophic failure you would want a log for. And when data corrupts, it's generally easier to extract some degree of data from a text file than from a binary. Maybe as a log archive format, though...

michaelcampbell 10y ago
Ah, systemd.
- vacri 10y ago
  Don't worry, you can just have journald send loglines across the network to a centralised logserver... ... oh, wait, that hasn't been solved yet. Instead, there's a variety of hacky workarounds (send it to rsyslog and get it to ship; follow a journal with ncat and ship that; some others). Centralising journald logs is my current ops problem de juor.
  - simcop2387 10y ago
    yea that's the one problem I've seen with journald that needs a proper solution. The other features it brings I've loved for my desktop but I wouldn't want it for long term data/logging yet.
  - duncan_bayne 10y ago
    "Those who do not understand UNIX are condemned to reinvent it, poorly" - Henry Spencer
    - hackits 10y ago
      Does that imply that LINUX was implemented poorly? I allays found the concept of socket file's to be mysterious things (coming from a windows background)
      - pjc50 10y ago
        Socket file descriptors were in UNIX before Linux was written. They were eventually copied into Windows via "Winsock", and Windows shares the same basic idea of using ReadFile to access the sockets. (The "reinvent it badly" is systemd replacing syslogd without re-implementing functionality that was important to a big subset of admins, because the people who wrote systemd are focused on the personal workstation case to the exclusion of all others)
        duncan_bayne 10y ago
        But more fundamentally than the desktop / server focus, they either don't understand or don't value what makes UNIX great. systemd is in many respects a major departure from UNIX philosophy, and I predict it will mark an inflection point in the quality and usability of Linux distros that adopt it, hence my efforts to migrate my own systems to FreeBSD. For those wondering what I'm banging on about, please read The Art of UNIX Programming, which should have been called The Philosophy of UNIX: http://www.catb.org/esr/writings/taoup/html/ http://www.catb.org/esr/writings/taoup/html/. Edit: what I mean by fundamental is that it shouldn't matter whether the tools are intended for desktop or server use; they should be designed to interoperate seamlessly using text protocols via pipes, sockets and files. That way they can be composed, filtered and transformed in ways not yet dreamed of by their creators. The systemd folks are falling into the Microsoft and Apple trap by trying to anticipate how their software will be used, instead of building it so it can easily be hacked upon for uses they themselves haven't dreamed of (which oddly seems to include 'servers'). To put it bluntly: systemd has neither the hacker nature nor the UNIX nature, and history has been unkind to OSs with neither. I'm betting against it being a Good Thing in the long run.
        qwertyuiop924 10y ago
        Yeah. As much as I hate ESR (I hate to beat that dead horse, but I don't want anyone getting the impression that I like him) he got it right here in this case, and Systemd definitely does the Wrong Thing. It's not a Good Thing now, though: It's an attack surface.
  - ecnahc515 10y ago
    Look at systemd-journal-upload. Also look at systemd-journal-remote. I was able to get upload working with the journal-remote and fluentd's TCP input.
- qwertyuiop924 10y ago
  My thoughts exactly.
simcop2387 10y ago
Big problem with text logs is the moment something you log doesn't fit what you were expecting. A message has a new-line? all the sudden you either have to escape characters or you have to handle the data being put on the next line rather than the next log message. How do you detect when that happens? A message doesn't fit the format properly? Ok so let's encode the data, base64? now you can't grep the logs anymore for information, it's an opaque format with meta data, might as well use SQLite or some other structured format.
- the_duke 10y ago
  As someone else said, JSON is great for logging. Newlines get escaped, you just write line seperated json entries to a log file. JSON can be easily read by humans, and is trivial to parse in most programming languages. I like SQLite, but I don't think logging is a good use case.
  - SFJulie 10y ago
    except if one line of JSON gets corrupted ... all your log storage does go to the trash. Heard about people login JSON ? (that may be truncated because log lines can be truncated when write is called with size > PIP_BUF in a concurrent environment?) I know JSON is the new XML, but dinosaurs like me have learnt the hard way that logging should not be stored as a document but a journal of chunks considered truncated, and that relying on the atomicity of system read/write/close/open/seek/tell/unlink is a damned good idea because the day you need logs, is usually the day a major crash happens. Hence a day where a corruption of logs is more likely to happen. True though that when you have no crashs, you love JSON format. But even truer it is when an incident happens you want a resilient system that still can log in a reliable way.
    - int_19h 10y ago
      The log format, as described above, is one JSON object per line. So if one line gets corrupted, this only affects that one line. And this is possible because the only place where a newline can appear in JSON is whitespace (where it can be safely replaced with a space; although most JSON serializers provide enough control over the output that you can just make sure that it never appears there in the first place).

michaelcampbell 10y ago

Ah, systemd.

vacri 10y ago
Don't worry, you can just have journald send loglines across the network to a centralised logserver... ... oh, wait, that hasn't been solved yet. Instead, there's a variety of hacky workarounds (send it to rsyslog and get it to ship; follow a journal with ncat and ship that; some others). Centralising journald logs is my current ops problem de juor.
- simcop2387 10y ago
  yea that's the one problem I've seen with journald that needs a proper solution. The other features it brings I've loved for my desktop but I wouldn't want it for long term data/logging yet.
- duncan_bayne 10y ago
  "Those who do not understand UNIX are condemned to reinvent it, poorly" - Henry Spencer
  - hackits 10y ago
    Does that imply that LINUX was implemented poorly? I allays found the concept of socket file's to be mysterious things (coming from a windows background)
    - pjc50 10y ago
      Socket file descriptors were in UNIX before Linux was written. They were eventually copied into Windows via "Winsock", and Windows shares the same basic idea of using ReadFile to access the sockets. (The "reinvent it badly" is systemd replacing syslogd without re-implementing functionality that was important to a big subset of admins, because the people who wrote systemd are focused on the personal workstation case to the exclusion of all others)
      - duncan_bayne 10y ago
        But more fundamentally than the desktop / server focus, they either don't understand or don't value what makes UNIX great. systemd is in many respects a major departure from UNIX philosophy, and I predict it will mark an inflection point in the quality and usability of Linux distros that adopt it, hence my efforts to migrate my own systems to FreeBSD. For those wondering what I'm banging on about, please read The Art of UNIX Programming, which should have been called The Philosophy of UNIX: http://www.catb.org/esr/writings/taoup/html/ http://www.catb.org/esr/writings/taoup/html/. Edit: what I mean by fundamental is that it shouldn't matter whether the tools are intended for desktop or server use; they should be designed to interoperate seamlessly using text protocols via pipes, sockets and files. That way they can be composed, filtered and transformed in ways not yet dreamed of by their creators. The systemd folks are falling into the Microsoft and Apple trap by trying to anticipate how their software will be used, instead of building it so it can easily be hacked upon for uses they themselves haven't dreamed of (which oddly seems to include 'servers'). To put it bluntly: systemd has neither the hacker nature nor the UNIX nature, and history has been unkind to OSs with neither. I'm betting against it being a Good Thing in the long run.
        qwertyuiop924 10y ago
        Yeah. As much as I hate ESR (I hate to beat that dead horse, but I don't want anyone getting the impression that I like him) he got it right here in this case, and Systemd definitely does the Wrong Thing. It's not a Good Thing now, though: It's an attack surface.
- ecnahc515 10y ago
  Look at systemd-journal-upload. Also look at systemd-journal-remote. I was able to get upload working with the journal-remote and fluentd's TCP input.
qwertyuiop924 10y ago
My thoughts exactly.

simcop2387 10y ago

Big problem with text logs is the moment something you log doesn't fit what you were expecting. A message has a new-line? all the sudden you either have to escape characters or you have to handle the data being put on the next line rather than the next log message. How do you detect when that happens? A message doesn't fit the format properly? Ok so let's encode the data, base64? now you can't grep the logs anymore for information, it's an opaque format with meta data, might as well use SQLite or some other structured format.

the_duke 10y ago
As someone else said, JSON is great for logging. Newlines get escaped, you just write line seperated json entries to a log file. JSON can be easily read by humans, and is trivial to parse in most programming languages. I like SQLite, but I don't think logging is a good use case.
- SFJulie 10y ago
  except if one line of JSON gets corrupted ... all your log storage does go to the trash. Heard about people login JSON ? (that may be truncated because log lines can be truncated when write is called with size > PIP_BUF in a concurrent environment?) I know JSON is the new XML, but dinosaurs like me have learnt the hard way that logging should not be stored as a document but a journal of chunks considered truncated, and that relying on the atomicity of system read/write/close/open/seek/tell/unlink is a damned good idea because the day you need logs, is usually the day a major crash happens. Hence a day where a corruption of logs is more likely to happen. True though that when you have no crashs, you love JSON format. But even truer it is when an incident happens you want a resilient system that still can log in a reliable way.
  - int_19h 10y ago
    The log format, as described above, is one JSON object per line. So if one line gets corrupted, this only affects that one line. And this is possible because the only place where a newline can appear in JSON is whitespace (where it can be safely replaced with a space; although most JSON serializers provide enough control over the output that you can just make sure that it never appears there in the first place).