4 ms·

Logs Are Streams, Not Files (2011)

52 points by fbuilesv 12y ago

farva 12y ago
It's not like there's really a difference between the two, under *nix.
- kiyoto 12y ago
  I see what you are getting at, but my experience as one of the "mailing list" people for Fluentd (an open source log collector) has been a bit different, especially when a lot of data is written to a file rapidly: For example, when you try to tail a log file with log rotation reliably, you run into all sorts of edge cases. Such issues shouldn't exist if files and streams were truly interchangeable in the context of logging.
  - antocv 12y ago
    > when you try to tail a log file with log rotation reliably, Then you do tail --follow -f yourfile. Files and streams are the same thing.
    - bluefinity 12y ago
      I think you mean tail -F yourfile. --follow and -f are the same thing.
      - antocv 12y ago
        No, I meant --follow=name because the default is descriptor. -f is short for --follow=descriptor
  - tszming 12y ago
    What issue you are experiencing with fluentd in_tail plugin? (As it already support log rotation)
    - kiyoto 12y ago
      Sorry that my point wasn't clear: it's not so much that I experience issues with in_tail. in_tail is actually solid at detecting log rotation, etc. My point was this is not a trivial problem, and a lot of the code inside in_tail.rb is for detecting and handling log rotation gracefully.
- jasode 12y ago
  Yes, but you're using a technical definition of FILE and STREAM (pipes).[1] However, the author was talking about a conceptual difference instead of the technical one. He's recommending that people think of logs as streams instead of files (again, "files" as an abstraction concept and not as FILE* as Unix o/s descriptor). The shift in thinking would trigger a different approach to writing log data. In any case, I think the Jay Kreps article on logs has much more information -- especially with distributed systems.[2] [1](E.g. The C Language fopen() returns a FILE* pointer and can open an actual file on disk or a transient named pipe (mkfifo)). [2]http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying http://engineering.linkedin.com/distributed-systems/log-what...
  - antocv 12y ago
    People should educate themselves and stop thinking of files as, whateveer it is that makes them think that logfiles are bad. Do they seriously think a file is like a magical stone being dropped where ever it may be? A self-contained unit, like a bit? Well, they're wrong. Files are streams of bits. Streams of bits can be in transition or stationary. Example, moving bits from here to there, over network or on the harddrive - bits in transition, a collection called a file. A bunch of bits on a hard-drive or in RAM, is still a stream, but not in the process of being copied/moved elsewhere.
philsnow 12y ago
sink/drain, not source/sink ? Does anybody use "sink" to mean the place where stuff comes out of (from a particular system's perspective) rather than the place where stuff goes ?
- gobengo 12y ago
  nope
- stinos 12y ago
  last 'streaming' applications we wrote, some for audio processing, others for image processing, we used the source/processor/sink notion (didn't even think of using drain) - source = origin of data, sink = destination fordata, processor = data goes in and comes out again, so basically sink at input end and source at output end. But I don't think I ever saw sink to be used as a source of data.
colmmacc 12y ago
From the article: > a better conceptual model is to treat logs as time-ordered streams At scale it's probably better still to re-think logs as weakly-ordered lossy streams. One form of weak-ordering is the inevitable jitter that comes with having multiple processes, threads or machines; without some kind of global lock (which would be impactful to performance) it stops being possible to have a true before/after relationship between individual log entries. Another form of weak ordering is that it's very common for log entries to be recorded only at the end of an operation, irrespective of its duration; so a single instantaneous entry really represents a time-span of activity with all sorts of fuzzy before/after/concurrent-to relationships to other entries. But maybe the most overlooked kind of weak ordering is one that is rarely found in logging systems, but is highly desirable: log streams should ideally be processed in LIFO order. If you're building some kind of analytical or visualisation system or near real-time processor for log data, you care most about "now". Inevitably there are processing queues and batches and so on to deal with; but practically every logging system just orders the entries by "time" and handles those queues as FIFO. If a backlog arises; you must wait for the old data to process before seeing the new. Change these queues and batching systems to LIFOs and you get really powerful behavior; recent data always takes priority but you can still backfill historical gaps. Unix files are particularly poorly suited to this pattern though - even though a stack is a simple data-structure, it's not something that you can easily emulate with a file-system and command line tools.
- justincormack 12y ago
  You could write some simple code that seeks if there is sufficient data between the file position and the end and puts the gap on a queue for later processing. You need to be able to find end of record markers but that's it.
- jimmaswell 12y ago
  >it stops being possible to have a true before/after relationship between individual log entries. Timestamps solve this easily. Synchronizing the time between machines isn't hard
  - jasode 12y ago
antocv 12y ago
Files are streams.
falcolas 12y ago
> Programs that send their logs directly to a logfile lose all the power and flexibility of unix streams. That's because if they push their data to stdout, and it's not caught by a pipe, the program will halt when the OS stdout buffer is filled. > How many programs end up re-writing log rotation, for example? This one is because files over a certain size cause certain file management systems, or kernels, to break logging. If they didn't rotate the files, the system would become unresponsive at worst, or the program would go down at best. Plus, if you take care of rotation and compression yourself (either directly or through a logrotated conf), you don't have to worry about filling a disk & causing an outage. In short, logging is hard, because systems are managed by people. And people rarely get the logging setups right the first time.

52 points by fbuilesv 12y ago

farva 12y ago

It's not like there's really a difference between the two, under *nix.

kiyoto 12y ago
I see what you are getting at, but my experience as one of the "mailing list" people for Fluentd (an open source log collector) has been a bit different, especially when a lot of data is written to a file rapidly: For example, when you try to tail a log file with log rotation reliably, you run into all sorts of edge cases. Such issues shouldn't exist if files and streams were truly interchangeable in the context of logging.
- antocv 12y ago
  > when you try to tail a log file with log rotation reliably, Then you do tail --follow -f yourfile. Files and streams are the same thing.
  - bluefinity 12y ago
    I think you mean tail -F yourfile. --follow and -f are the same thing.
    - antocv 12y ago
      No, I meant --follow=name because the default is descriptor. -f is short for --follow=descriptor
- tszming 12y ago
  What issue you are experiencing with fluentd in_tail plugin? (As it already support log rotation)
  - kiyoto 12y ago
    Sorry that my point wasn't clear: it's not so much that I experience issues with in_tail. in_tail is actually solid at detecting log rotation, etc. My point was this is not a trivial problem, and a lot of the code inside in_tail.rb is for detecting and handling log rotation gracefully.
jasode 12y ago
Yes, but you're using a technical definition of FILE and STREAM (pipes).[1] However, the author was talking about a conceptual difference instead of the technical one. He's recommending that people think of logs as streams instead of files (again, "files" as an abstraction concept and not as FILE* as Unix o/s descriptor). The shift in thinking would trigger a different approach to writing log data. In any case, I think the Jay Kreps article on logs has much more information -- especially with distributed systems.[2] [1](E.g. The C Language fopen() returns a FILE* pointer and can open an actual file on disk or a transient named pipe (mkfifo)). [2]http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying http://engineering.linkedin.com/distributed-systems/log-what...
- antocv 12y ago
  People should educate themselves and stop thinking of files as, whateveer it is that makes them think that logfiles are bad. Do they seriously think a file is like a magical stone being dropped where ever it may be? A self-contained unit, like a bit? Well, they're wrong. Files are streams of bits. Streams of bits can be in transition or stationary. Example, moving bits from here to there, over network or on the harddrive - bits in transition, a collection called a file. A bunch of bits on a hard-drive or in RAM, is still a stream, but not in the process of being copied/moved elsewhere.

philsnow 12y ago

sink/drain, not source/sink ? Does anybody use "sink" to mean the place where stuff comes out of (from a particular system's perspective) rather than the place where stuff goes ?

gobengo 12y ago
nope
stinos 12y ago
last 'streaming' applications we wrote, some for audio processing, others for image processing, we used the source/processor/sink notion (didn't even think of using drain) - source = origin of data, sink = destination fordata, processor = data goes in and comes out again, so basically sink at input end and source at output end. But I don't think I ever saw sink to be used as a source of data.

colmmacc 12y ago

From the article: > a better conceptual model is to treat logs as time-ordered streams At scale it's probably better still to re-think logs as weakly-ordered lossy streams. One form of weak-ordering is the inevitable jitter that comes with having multiple processes, threads or machines; without some kind of global lock (which would be impactful to performance) it stops being possible to have a true before/after relationship between individual log entries. Another form of weak ordering is that it's very common for log entries to be recorded only at the end of an operation, irrespective of its duration; so a single instantaneous entry really represents a time-span of activity with all sorts of fuzzy before/after/concurrent-to relationships to other entries. But maybe the most overlooked kind of weak ordering is one that is rarely found in logging systems, but is highly desirable: log streams should ideally be processed in LIFO order. If you're building some kind of analytical or visualisation system or near real-time processor for log data, you care most about "now". Inevitably there are processing queues and batches and so on to deal with; but practically every logging system just orders the entries by "time" and handles those queues as FIFO. If a backlog arises; you must wait for the old data to process before seeing the new. Change these queues and batching systems to LIFOs and you get really powerful behavior; recent data always takes priority but you can still backfill historical gaps. Unix files are particularly poorly suited to this pattern though - even though a stack is a simple data-structure, it's not something that you can easily emulate with a file-system and command line tools.

justincormack 12y ago
You could write some simple code that seeks if there is sufficient data between the file position and the end and puts the gap on a queue for later processing. You need to be able to find end of record markers but that's it.
jimmaswell 12y ago
>it stops being possible to have a true before/after relationship between individual log entries. Timestamps solve this easily. Synchronizing the time between machines isn't hard
- jasode 12y ago

antocv 12y ago

Files are streams.

falcolas 12y ago

> Programs that send their logs directly to a logfile lose all the power and flexibility of unix streams. That's because if they push their data to stdout, and it's not caught by a pipe, the program will halt when the OS stdout buffer is filled. > How many programs end up re-writing log rotation, for example? This one is because files over a certain size cause certain file management systems, or kernels, to break logging. If they didn't rotate the files, the system would become unresponsive at worst, or the program would go down at best. Plus, if you take care of rotation and compression yourself (either directly or through a logrotated conf), you don't have to worry about filling a disk & causing an outage. In short, logging is hard, because systems are managed by people. And people rarely get the logging setups right the first time.