5 ms·

It's not like there's really a difference between the two, under *nix.

by farva 12y ago

kiyoto 12y ago
I see what you are getting at, but my experience as one of the "mailing list" people for Fluentd (an open source log collector) has been a bit different, especially when a lot of data is written to a file rapidly: For example, when you try to tail a log file with log rotation reliably, you run into all sorts of edge cases. Such issues shouldn't exist if files and streams were truly interchangeable in the context of logging.
- antocv 12y ago
  > when you try to tail a log file with log rotation reliably, Then you do tail --follow -f yourfile. Files and streams are the same thing.
  - bluefinity 12y ago
    I think you mean tail -F yourfile. --follow and -f are the same thing.
    - antocv 12y ago
      No, I meant --follow=name because the default is descriptor. -f is short for --follow=descriptor
- tszming 12y ago
  What issue you are experiencing with fluentd in_tail plugin? (As it already support log rotation)
  - kiyoto 12y ago
    Sorry that my point wasn't clear: it's not so much that I experience issues with in_tail. in_tail is actually solid at detecting log rotation, etc. My point was this is not a trivial problem, and a lot of the code inside in_tail.rb is for detecting and handling log rotation gracefully.
jasode 12y ago
Yes, but you're using a technical definition of FILE and STREAM (pipes).[1] However, the author was talking about a conceptual difference instead of the technical one. He's recommending that people think of logs as streams instead of files (again, "files" as an abstraction concept and not as FILE* as Unix o/s descriptor). The shift in thinking would trigger a different approach to writing log data. In any case, I think the Jay Kreps article on logs has much more information -- especially with distributed systems.[2] [1](E.g. The C Language fopen() returns a FILE* pointer and can open an actual file on disk or a transient named pipe (mkfifo)). [2]http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying http://engineering.linkedin.com/distributed-systems/log-what...
- antocv 12y ago
  People should educate themselves and stop thinking of files as, whateveer it is that makes them think that logfiles are bad. Do they seriously think a file is like a magical stone being dropped where ever it may be? A self-contained unit, like a bit? Well, they're wrong. Files are streams of bits. Streams of bits can be in transition or stationary. Example, moving bits from here to there, over network or on the harddrive - bits in transition, a collection called a file. A bunch of bits on a hard-drive or in RAM, is still a stream, but not in the process of being copied/moved elsewhere.

It's not like there's really a difference between the two, under *nix.

by farva 12y ago

It's not like there's really a difference between the two, under *nix.

I see what you are getting at, but my experience as one of the "mailing list" people for Fluentd (an open source log collector) has been a bit different, especially when a lot of data is written to a file rapidly: For example, when you try to tail a log file with log rotation reliably, you run into all sorts of edge cases. Such issues shouldn't exist if files and streams were truly interchangeable in the context of logging.

antocv 12y ago
> when you try to tail a log file with log rotation reliably, Then you do tail --follow -f yourfile. Files and streams are the same thing.
- bluefinity 12y ago
  I think you mean tail -F yourfile. --follow and -f are the same thing.
  - antocv 12y ago
    No, I meant --follow=name because the default is descriptor. -f is short for --follow=descriptor
tszming 12y ago
What issue you are experiencing with fluentd in_tail plugin? (As it already support log rotation)
- kiyoto 12y ago
  Sorry that my point wasn't clear: it's not so much that I experience issues with in_tail. in_tail is actually solid at detecting log rotation, etc. My point was this is not a trivial problem, and a lot of the code inside in_tail.rb is for detecting and handling log rotation gracefully.

jasode 12y ago

Yes, but you're using a technical definition of FILE and STREAM (pipes).[1] However, the author was talking about a conceptual difference instead of the technical one. He's recommending that people think of logs as streams instead of files (again, "files" as an abstraction concept and not as FILE* as Unix o/s descriptor). The shift in thinking would trigger a different approach to writing log data. In any case, I think the Jay Kreps article on logs has much more information -- especially with distributed systems.[2] [1](E.g. The C Language fopen() returns a FILE* pointer and can open an actual file on disk or a transient named pipe (mkfifo)). [2]http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying http://engineering.linkedin.com/distributed-systems/log-what...

antocv 12y ago
People should educate themselves and stop thinking of files as, whateveer it is that makes them think that logfiles are bad. Do they seriously think a file is like a magical stone being dropped where ever it may be? A self-contained unit, like a bit? Well, they're wrong. Files are streams of bits. Streams of bits can be in transition or stationary. Example, moving bits from here to there, over network or on the harddrive - bits in transition, a collection called a file. A bunch of bits on a hard-drive or in RAM, is still a stream, but not in the process of being copied/moved elsewhere.