4 ms·

No, writing files is nearly as bad as writing to stdout -- both imply someone else has to parse the output. If the logs matter -- write them to an API that ret

by cduzz 3y ago

No, writing files is nearly as bad as writing to stdout -- both imply someone else has to parse the output. If the logs matter -- write them to an API that retains log structure by serializing it. If the logs don't matter, then they don't matter, but typically the logs are used for all sorts of analytics processes after the fact. Asking someone else to parse your data is rude.

nightpool 3y ago
Again, "this is a case where 12 factor is so successful / dominant today I don't think you even know what it's arguing against anymore"
- cduzz 3y ago
  The twelve factor app says: A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout. During local development, the developer will view this stream in the foreground of their terminal to observe the app’s behavior. That guidance is deeply foolish; I have to assume the rest of the guidance is just as silly for anything but a toy application.
  - lexicality 3y ago
    Where does it say "unformatted"? All it says is that your logs should be sent to stdout. You write your logs in whatever structure you want and have something in the runtime environment (k8s, systemd, file redirection in bash whatever) forward them to the eventual storage location. I don't see the issue.
    - cduzz 3y ago
      So -- there's a file, that's created by some something in a container emitting a string of bytes. A file must go through a parsing and serialization phase to be useful to a logging infrastructure. Just because you decorate the contents of that file with { and } and other punctuation doesn't make that file a string of json records, until something's read and verified it. I'm supposed to just load it into memory hoping it's json or avro or whatever? Nope, errors happen; there's a limit to the number of bytes that can be sent from one place to another before the kernel may interrupt it (you're writing an application log and your runtime emits it's own debugging message; they get intermingled because they're both being written to a fifo). Maybe this is fine (usually it's fine) but eventually it isn't fine. Use a logging library, categories your logs, maybe start off with stdout but eventually put on some big person pants and use a logging api and use the stderr/stdout for high cardinality exceptions, not "all my logs"
      - jameshart 3y ago
        12 factor does not say ‘don’t use a logging library’ Use a logging library. Configure it to output well formed and categorized log messages to stdout. Let the runtime environment take that stream of data from there.
        cduzz 3y ago
        It certainly doesn't say "use a logging library" it says: A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout. During local development, the developer will view this stream in the foreground of their terminal to observe the app’s behavior. In staging or production deploys, each process’ stream will be captured by the execution environment, collated together with all other streams from the app, and routed to one or more final destinations for viewing and long-term archival. These archival destinations are not visible to or configurable by the app, and instead are completely managed by the execution environment. Open-source log routers (such as Logplex and Fluentd) are available for this purpose. The event stream for an app can be routed to a file, or watched via realtime tail in a terminal. Most significantly, the stream can be sent to a log indexing and analysis system such as Splunk, or a general-purpose data warehousing system such as Hadoop/Hive. These systems allow for great power and flexibility for introspecting an app’s behavior over time, including: This is, at scale, utter horseshit. If it said "do what's expedient, but use a library and be flexible in how you generate logs, but for goodness sake use any of the publicly available libraries to be expedient now and reliable and flexible later" I'd shrug and move on, but it's just absolute garbage. "Do a shitty job and someone else will fix it for you" is not good advice, even if it lets you get to market quickly.
        lmm 3y ago
        > It certainly doesn't say "use a logging library" Right. It's not telling you anything about whether to use a library or not. All it's talking about is where you output your logging streams to, not about how you produce or format those streams. Is it bad advice because it doesn't tell you you should wear sunscreen when you go outside in summer? (FWIW I haven't seen any logging library add enough value to be worthwhile. Eventually graduating to structured event logs is a good idea, but having started with a logging library doesn't actually get you any closer to that).
  - jameshart 3y ago
    What 12 factor is telling you to do is to not build a log file rotator into your application. It is the least controversial and most broadly beneficial of the 12 factors. Why do you think it’s foolish?
    - cduzz 3y ago
      It says "send it to stdout and let someone else figure it out." As that someone else, who often has bug reports like "why aren't my log events all in one record in my thing? Because you filled your stdout with newlines!" "Why did my message vanish? It was important!" "because it's malformed json because your runtime had an error; it's over there instead of over here." Programming's a stack of details; all of which matter. Log using a logging API where available; use stdout/stderr for exception logging only, if at all possible. If you want your logging to be useful, think carefully about how and what you log.
      - jameshart 3y ago
        That’s a total misreading of what 12 factor is about. All it means is to let someone else (other than the program itself) figure out how to get the log messages (whatever you decide they should be) from your process (running wherever it is) to wherever you want to have those logs for inspection. Do not hardcode into your application the assumption that they are written to a file or sent to a port. In your application, write them to stdout. In local dev, stdout will probably go to an interactive console. In a container running in a cloud hosted cluster it will probably go to a networked log aggregator that collects it all into an indexed time oriented data store along with other logs from other running instances. Because you used stdout, that’s easy to do. If you wrote them to a file whose name your application manages and changes itself to manage log rotation… either of those is hard to do.

by cduzz 3y ago

nightpool 3y ago

Again, "this is a case where 12 factor is so successful / dominant today I don't think you even know what it's arguing against anymore"

cduzz 3y ago
The twelve factor app says: A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout. During local development, the developer will view this stream in the foreground of their terminal to observe the app’s behavior. That guidance is deeply foolish; I have to assume the rest of the guidance is just as silly for anything but a toy application.
- lexicality 3y ago
  Where does it say "unformatted"? All it says is that your logs should be sent to stdout. You write your logs in whatever structure you want and have something in the runtime environment (k8s, systemd, file redirection in bash whatever) forward them to the eventual storage location. I don't see the issue.
  - cduzz 3y ago
    So -- there's a file, that's created by some something in a container emitting a string of bytes. A file must go through a parsing and serialization phase to be useful to a logging infrastructure. Just because you decorate the contents of that file with { and } and other punctuation doesn't make that file a string of json records, until something's read and verified it. I'm supposed to just load it into memory hoping it's json or avro or whatever? Nope, errors happen; there's a limit to the number of bytes that can be sent from one place to another before the kernel may interrupt it (you're writing an application log and your runtime emits it's own debugging message; they get intermingled because they're both being written to a fifo). Maybe this is fine (usually it's fine) but eventually it isn't fine. Use a logging library, categories your logs, maybe start off with stdout but eventually put on some big person pants and use a logging api and use the stderr/stdout for high cardinality exceptions, not "all my logs"
    - jameshart 3y ago
      12 factor does not say ‘don’t use a logging library’ Use a logging library. Configure it to output well formed and categorized log messages to stdout. Let the runtime environment take that stream of data from there.
      - cduzz 3y ago
        It certainly doesn't say "use a logging library" it says: A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout. During local development, the developer will view this stream in the foreground of their terminal to observe the app’s behavior. In staging or production deploys, each process’ stream will be captured by the execution environment, collated together with all other streams from the app, and routed to one or more final destinations for viewing and long-term archival. These archival destinations are not visible to or configurable by the app, and instead are completely managed by the execution environment. Open-source log routers (such as Logplex and Fluentd) are available for this purpose. The event stream for an app can be routed to a file, or watched via realtime tail in a terminal. Most significantly, the stream can be sent to a log indexing and analysis system such as Splunk, or a general-purpose data warehousing system such as Hadoop/Hive. These systems allow for great power and flexibility for introspecting an app’s behavior over time, including: This is, at scale, utter horseshit. If it said "do what's expedient, but use a library and be flexible in how you generate logs, but for goodness sake use any of the publicly available libraries to be expedient now and reliable and flexible later" I'd shrug and move on, but it's just absolute garbage. "Do a shitty job and someone else will fix it for you" is not good advice, even if it lets you get to market quickly.
        lmm 3y ago
        > It certainly doesn't say "use a logging library" Right. It's not telling you anything about whether to use a library or not. All it's talking about is where you output your logging streams to, not about how you produce or format those streams. Is it bad advice because it doesn't tell you you should wear sunscreen when you go outside in summer? (FWIW I haven't seen any logging library add enough value to be worthwhile. Eventually graduating to structured event logs is a good idea, but having started with a logging library doesn't actually get you any closer to that).
- jameshart 3y ago
  What 12 factor is telling you to do is to not build a log file rotator into your application. It is the least controversial and most broadly beneficial of the 12 factors. Why do you think it’s foolish?
  - cduzz 3y ago
    It says "send it to stdout and let someone else figure it out." As that someone else, who often has bug reports like "why aren't my log events all in one record in my thing? Because you filled your stdout with newlines!" "Why did my message vanish? It was important!" "because it's malformed json because your runtime had an error; it's over there instead of over here." Programming's a stack of details; all of which matter. Log using a logging API where available; use stdout/stderr for exception logging only, if at all possible. If you want your logging to be useful, think carefully about how and what you log.
    - jameshart 3y ago
      That’s a total misreading of what 12 factor is about. All it means is to let someone else (other than the program itself) figure out how to get the log messages (whatever you decide they should be) from your process (running wherever it is) to wherever you want to have those logs for inspection. Do not hardcode into your application the assumption that they are written to a file or sent to a port. In your application, write them to stdout. In local dev, stdout will probably go to an interactive console. In a container running in a cloud hosted cluster it will probably go to a networked log aggregator that collects it all into an indexed time oriented data store along with other logs from other running instances. Because you used stdout, that’s easy to do. If you wrote them to a file whose name your application manages and changes itself to manage log rotation… either of those is hard to do.