5 ms·

A little light on detail. Can someone please explain exactly how this works? The best I can figure is it's a shipper replacement. So apps write to a log file a

A little light on detail. Can someone please explain exactly how this works? The best I can figure is it's a shipper replacement. So apps write to a log file as usual, but it's actually buffered in memory and pushed to a central server via a FIFO queue. Given it's all in-memory, it will be small and transient, so you're completely relying on the central server to store it reliably. (Not a criticism, just trying to understand it.)

mmastrac 12y ago
My first impression is that this is a fantastic and novel development for centralized log analysis. Basically, it's a virtual filesystem that pretends to create files, but actually intercepts writes to log-style files in this virtual location and buffers those in memory, eventually shipping them off to a place where you can run central analytics on them (ie: Splunk, Logstash, etc). The application doesn't know that it's writing to a virtual file: it just opens a descriptor, dumps a line or two every few seconds and continues along its merry way. The logs never touch the disk, which means that they don't content for limited disk I/O bandwidth.
- ams6110 12y ago
  How many logging frameworks don't already support syslog?
  - e12e 12y ago
    Probably none, given: "echo hi|logger".
    - Rantenki 12y ago
      Take with a grain of salt, since it's anecdotal, but I have seen logger hang when syslog rotates. Logger is really only 100% reliable when you are piping it the output of a command that runs in a bounded amount of time. If you have a real daemon it's always better to use syslog directly, and an inability to use syslog is maybe an indication of the quality of the daemon in question.
- achillean 12y ago
  I actually wrote exactly the same thing years ago (2007), even had the same name :) http://sourceforge.net/projects/loggerfs/ http://sourceforge.net/projects/loggerfs/
  - seanewest 12y ago
    Put it on github -- people might send some Pull Requests.
- derefr 12y ago
  Presuming your daemons rotate their logs, what's the benefit of this over mounting a plain-old tmpfs over your logs folder? Either way, the shipper reads virtual files that are actually stored in memory, and which eventually get purged. It's just the thresholds that are different.
  - mmastrac 12y ago
    You'd have to be rolling your logs fairly often to avoid filling up tmpfs with log lines that have already been shipped. The advantage of this approach is that log lines that are shipped are no longer on the box at all.
deleted 12y ago
[deleted]

mmastrac 12y ago

My first impression is that this is a fantastic and novel development for centralized log analysis. Basically, it's a virtual filesystem that pretends to create files, but actually intercepts writes to log-style files in this virtual location and buffers those in memory, eventually shipping them off to a place where you can run central analytics on them (ie: Splunk, Logstash, etc). The application doesn't know that it's writing to a virtual file: it just opens a descriptor, dumps a line or two every few seconds and continues along its merry way. The logs never touch the disk, which means that they don't content for limited disk I/O bandwidth.

ams6110 12y ago
How many logging frameworks don't already support syslog?
- e12e 12y ago
  Probably none, given: "echo hi|logger".
  - Rantenki 12y ago
    Take with a grain of salt, since it's anecdotal, but I have seen logger hang when syslog rotates. Logger is really only 100% reliable when you are piping it the output of a command that runs in a bounded amount of time. If you have a real daemon it's always better to use syslog directly, and an inability to use syslog is maybe an indication of the quality of the daemon in question.
achillean 12y ago
I actually wrote exactly the same thing years ago (2007), even had the same name :) http://sourceforge.net/projects/loggerfs/ http://sourceforge.net/projects/loggerfs/
- seanewest 12y ago
  Put it on github -- people might send some Pull Requests.
derefr 12y ago
Presuming your daemons rotate their logs, what's the benefit of this over mounting a plain-old tmpfs over your logs folder? Either way, the shipper reads virtual files that are actually stored in memory, and which eventually get purged. It's just the thresholds that are different.
- mmastrac 12y ago
  You'd have to be rolling your logs fairly often to avoid filling up tmpfs with log lines that have already been shipped. The advantage of this approach is that log lines that are shipped are no longer on the box at all.

deleted 12y ago

[deleted]