6 ms·

I found it useful for logging. Logging tends to have structured data associated with it. Trying to re-parse log files into some meta data gets tiresome and is e

by hackits 10y ago

I found it useful for logging. Logging tends to have structured data associated with it. Trying to re-parse log files into some meta data gets tiresome and is error prone.

qwertyuiop924 10y ago
Honestly, I'd say that logging should be text. Yes, SQLite DBs are hard to corrupt, but it can happen, especially in the sort of catastrophic failure you would want a log for. And when data corrupts, it's generally easier to extract some degree of data from a text file than from a binary. Maybe as a log archive format, though...
- michaelcampbell 10y ago
  Ah, systemd.
  - vacri 10y ago
    Don't worry, you can just have journald send loglines across the network to a centralised logserver... ... oh, wait, that hasn't been solved yet. Instead, there's a variety of hacky workarounds (send it to rsyslog and get it to ship; follow a journal with ncat and ship that; some others). Centralising journald logs is my current ops problem de juor.
    - simcop2387 10y ago
      yea that's the one problem I've seen with journald that needs a proper solution. The other features it brings I've loved for my desktop but I wouldn't want it for long term data/logging yet.
    - duncan_bayne 10y ago
      "Those who do not understand UNIX are condemned to reinvent it, poorly" - Henry Spencer
      - hackits 10y ago
        Does that imply that LINUX was implemented poorly? I allays found the concept of socket file's to be mysterious things (coming from a windows background)
        pjc50 10y ago
        Socket file descriptors were in UNIX before Linux was written. They were eventually copied into Windows via "Winsock", and Windows shares the same basic idea of using ReadFile to access the sockets. (The "reinvent it badly" is systemd replacing syslogd without re-implementing functionality that was important to a big subset of admins, because the people who wrote systemd are focused on the personal workstation case to the exclusion of all others)
        duncan_bayne 10y ago
        But more fundamentally than the desktop / server focus, they either don't understand or don't value what makes UNIX great. systemd is in many respects a major departure from UNIX philosophy, and I predict it will mark an inflection point in the quality and usability of Linux distros that adopt it, hence my efforts to migrate my own systems to FreeBSD. For those wondering what I'm banging on about, please read The Art of UNIX Programming, which should have been called The Philosophy of UNIX: http://www.catb.org/esr/writings/taoup/html/ http://www.catb.org/esr/writings/taoup/html/. Edit: what I mean by fundamental is that it shouldn't matter whether the tools are intended for desktop or server use; they should be designed to interoperate seamlessly using text protocols via pipes, sockets and files. That way they can be composed, filtered and transformed in ways not yet dreamed of by their creators. The systemd folks are falling into the Microsoft and Apple trap by trying to anticipate how their software will be used, instead of building it so it can easily be hacked upon for uses they themselves haven't dreamed of (which oddly seems to include 'servers'). To put it bluntly: systemd has neither the hacker nature nor the UNIX nature, and history has been unkind to OSs with neither. I'm betting against it being a Good Thing in the long run.
        10y ago
    - ecnahc515 10y ago
      Look at systemd-journal-upload. Also look at systemd-journal-remote. I was able to get upload working with the journal-remote and fluentd's TCP input.
  - qwertyuiop924 10y ago
    My thoughts exactly.
- simcop2387 10y ago
  Big problem with text logs is the moment something you log doesn't fit what you were expecting. A message has a new-line? all the sudden you either have to escape characters or you have to handle the data being put on the next line rather than the next log message. How do you detect when that happens? A message doesn't fit the format properly? Ok so let's encode the data, base64? now you can't grep the logs anymore for information, it's an opaque format with meta data, might as well use SQLite or some other structured format.
  - the_duke 10y ago
    As someone else said, JSON is great for logging. Newlines get escaped, you just write line seperated json entries to a log file. JSON can be easily read by humans, and is trivial to parse in most programming languages. I like SQLite, but I don't think logging is a good use case.
    - SFJulie 10y ago
      except if one line of JSON gets corrupted ... all your log storage does go to the trash. Heard about people login JSON ? (that may be truncated because log lines can be truncated when write is called with size > PIP_BUF in a concurrent environment?) I know JSON is the new XML, but dinosaurs like me have learnt the hard way that logging should not be stored as a document but a journal of chunks considered truncated, and that relying on the atomicity of system read/write/close/open/seek/tell/unlink is a damned good idea because the day you need logs, is usually the day a major crash happens. Hence a day where a corruption of logs is more likely to happen. True though that when you have no crashs, you love JSON format. But even truer it is when an incident happens you want a resilient system that still can log in a reliable way.
      - int_19h 10y ago
        The log format, as described above, is one JSON object per line. So if one line gets corrupted, this only affects that one line. And this is possible because the only place where a newline can appear in JSON is whitespace (where it can be safely replaced with a space; although most JSON serializers provide enough control over the output that you can just make sure that it never appears there in the first place).
tracker1 10y ago
I'm a pretty big fan of line-separated JSON records... since JSON by default in most platforms serializes as a string, where linefeeds are escaped (\n) in strings, then you can separate records with a linefeed. This can be streamed and even gzipped in said stream, meaning you get compression and a pretty easy format you can use in most platforms these days with very little intermediate processing or extensions. With JSON you can have additional metadata on each record, and not have it affect the mainline... now, this cannot be queried directly, but it can be very easily streamed/imported elsewhere.
- sebcat 10y ago
  For certain tasks, I like to work on streams of JSON records too. jq(1) makes it really easy. However, requiring records to be line-delimited is a point of failure when accepting input from external sources. There should be no difference between: {"foo": 1}{"bar": 2} and {"foo": 1} {"bar": 2} or any variations of common whitespace between the objects. Just parse objects incrementally, one at a time. Most JSON libraries supports this. As for general logging: I prefer it to be structured text written to stderr, and have a PEG parsing it for me. stderr is pretty much a catch-all log stream, and some 3rd party software insist on writing to it (most do however offer a way to override this behavior, e.g., chromium-headless, libxml2, &c). Having a PEG for those cases means I can still get JSON for everything if I need to, even if the data in the log is unexpected. If I need to, I can then modify the PEG for that use case. Of course, it's a case-by-case, pragmatic decision. If I only want to log certain data and disregard stderr, or if I can guarantee everything that gets written to stderr will be valid JSON, writing logs as JSON may be better.
  - tracker1 10y ago
    It's simply easier to determine when an object ends with a delimiter... {"dtm":"2016-...",...,meta:{...}} That requires more complicated input processing/streaming than read until '\n' then parse the record...
    - eatonphil 10y ago
lultimouomo 10y ago
When I did this, I found that I needed to be pretty careful when rotating logs. The online backup API fits the need well, but if you're using a wrapper lib around SQLite you most probably won't be able to access it.

by hackits 10y ago

I found it useful for logging. Logging tends to have structured data associated with it. Trying to re-parse log files into some meta data gets tiresome and is error prone.

qwertyuiop924 10y ago

Honestly, I'd say that logging should be text. Yes, SQLite DBs are hard to corrupt, but it can happen, especially in the sort of catastrophic failure you would want a log for. And when data corrupts, it's generally easier to extract some degree of data from a text file than from a binary. Maybe as a log archive format, though...

michaelcampbell 10y ago
Ah, systemd.
- vacri 10y ago
  Don't worry, you can just have journald send loglines across the network to a centralised logserver... ... oh, wait, that hasn't been solved yet. Instead, there's a variety of hacky workarounds (send it to rsyslog and get it to ship; follow a journal with ncat and ship that; some others). Centralising journald logs is my current ops problem de juor.
  - simcop2387 10y ago
    yea that's the one problem I've seen with journald that needs a proper solution. The other features it brings I've loved for my desktop but I wouldn't want it for long term data/logging yet.
  - duncan_bayne 10y ago
    "Those who do not understand UNIX are condemned to reinvent it, poorly" - Henry Spencer
    - hackits 10y ago
      Does that imply that LINUX was implemented poorly? I allays found the concept of socket file's to be mysterious things (coming from a windows background)
      - pjc50 10y ago
        Socket file descriptors were in UNIX before Linux was written. They were eventually copied into Windows via "Winsock", and Windows shares the same basic idea of using ReadFile to access the sockets. (The "reinvent it badly" is systemd replacing syslogd without re-implementing functionality that was important to a big subset of admins, because the people who wrote systemd are focused on the personal workstation case to the exclusion of all others)
        duncan_bayne 10y ago
        But more fundamentally than the desktop / server focus, they either don't understand or don't value what makes UNIX great. systemd is in many respects a major departure from UNIX philosophy, and I predict it will mark an inflection point in the quality and usability of Linux distros that adopt it, hence my efforts to migrate my own systems to FreeBSD. For those wondering what I'm banging on about, please read The Art of UNIX Programming, which should have been called The Philosophy of UNIX: http://www.catb.org/esr/writings/taoup/html/ http://www.catb.org/esr/writings/taoup/html/. Edit: what I mean by fundamental is that it shouldn't matter whether the tools are intended for desktop or server use; they should be designed to interoperate seamlessly using text protocols via pipes, sockets and files. That way they can be composed, filtered and transformed in ways not yet dreamed of by their creators. The systemd folks are falling into the Microsoft and Apple trap by trying to anticipate how their software will be used, instead of building it so it can easily be hacked upon for uses they themselves haven't dreamed of (which oddly seems to include 'servers'). To put it bluntly: systemd has neither the hacker nature nor the UNIX nature, and history has been unkind to OSs with neither. I'm betting against it being a Good Thing in the long run.
        10y ago
  - ecnahc515 10y ago
    Look at systemd-journal-upload. Also look at systemd-journal-remote. I was able to get upload working with the journal-remote and fluentd's TCP input.
- qwertyuiop924 10y ago
  My thoughts exactly.
simcop2387 10y ago
Big problem with text logs is the moment something you log doesn't fit what you were expecting. A message has a new-line? all the sudden you either have to escape characters or you have to handle the data being put on the next line rather than the next log message. How do you detect when that happens? A message doesn't fit the format properly? Ok so let's encode the data, base64? now you can't grep the logs anymore for information, it's an opaque format with meta data, might as well use SQLite or some other structured format.
- the_duke 10y ago
  As someone else said, JSON is great for logging. Newlines get escaped, you just write line seperated json entries to a log file. JSON can be easily read by humans, and is trivial to parse in most programming languages. I like SQLite, but I don't think logging is a good use case.
  - SFJulie 10y ago
    except if one line of JSON gets corrupted ... all your log storage does go to the trash. Heard about people login JSON ? (that may be truncated because log lines can be truncated when write is called with size > PIP_BUF in a concurrent environment?) I know JSON is the new XML, but dinosaurs like me have learnt the hard way that logging should not be stored as a document but a journal of chunks considered truncated, and that relying on the atomicity of system read/write/close/open/seek/tell/unlink is a damned good idea because the day you need logs, is usually the day a major crash happens. Hence a day where a corruption of logs is more likely to happen. True though that when you have no crashs, you love JSON format. But even truer it is when an incident happens you want a resilient system that still can log in a reliable way.
    - int_19h 10y ago
      The log format, as described above, is one JSON object per line. So if one line gets corrupted, this only affects that one line. And this is possible because the only place where a newline can appear in JSON is whitespace (where it can be safely replaced with a space; although most JSON serializers provide enough control over the output that you can just make sure that it never appears there in the first place).

tracker1 10y ago

I'm a pretty big fan of line-separated JSON records... since JSON by default in most platforms serializes as a string, where linefeeds are escaped (\n) in strings, then you can separate records with a linefeed. This can be streamed and even gzipped in said stream, meaning you get compression and a pretty easy format you can use in most platforms these days with very little intermediate processing or extensions. With JSON you can have additional metadata on each record, and not have it affect the mainline... now, this cannot be queried directly, but it can be very easily streamed/imported elsewhere.

sebcat 10y ago
For certain tasks, I like to work on streams of JSON records too. jq(1) makes it really easy. However, requiring records to be line-delimited is a point of failure when accepting input from external sources. There should be no difference between: {"foo": 1}{"bar": 2} and {"foo": 1} {"bar": 2} or any variations of common whitespace between the objects. Just parse objects incrementally, one at a time. Most JSON libraries supports this. As for general logging: I prefer it to be structured text written to stderr, and have a PEG parsing it for me. stderr is pretty much a catch-all log stream, and some 3rd party software insist on writing to it (most do however offer a way to override this behavior, e.g., chromium-headless, libxml2, &c). Having a PEG for those cases means I can still get JSON for everything if I need to, even if the data in the log is unexpected. If I need to, I can then modify the PEG for that use case. Of course, it's a case-by-case, pragmatic decision. If I only want to log certain data and disregard stderr, or if I can guarantee everything that gets written to stderr will be valid JSON, writing logs as JSON may be better.
- tracker1 10y ago
  It's simply easier to determine when an object ends with a delimiter... {"dtm":"2016-...",...,meta:{...}} That requires more complicated input processing/streaming than read until '\n' then parse the record...
  - eatonphil 10y ago

lultimouomo 10y ago

When I did this, I found that I needed to be pretty careful when rotating logs. The online backup API fits the need well, but if you're using a wrapper lib around SQLite you most probably won't be able to access it.