4 ms·

One of the hardest things in ops is trying to see what the app is doing without benefit of the code to review. I know what the app does, I know the process flo

One of the hardest things in ops is trying to see what the app is doing without benefit of the code to review. I know what the app does, I know the process flows involved but 'failed to covert X at MyClass:44' doesn't have enough context for me. Also log levels - most developers over use warning and error levels and under uses debug and trace in my experience. It makes filtering out the noise harder. Rolling log files - roll on an event (new day, process bounced etc.) not on reaching a 10Mb limit or an equally arbitrary 1k lines limit. Use a header in the logfile, contain at least the version number of the code running, the user and host it is running on. Lastly stack traces, if they even have a use case at all in production, should not be in the main logfile.

nitrogen 14y ago
Lastly stack traces, if they even have a use case at all in production, should not be in the main logfile. I would have to disagree with this one. There's no way to generate the stack trace after the fact if a particular bug is extremely rare; having a complete call chain makes fixing the problem much easier. Someone once said that removing error handling once the code is working is like removing the landing gear once a plane is airborne.
Pfhreak 14y ago
I always use warning to indicate something actionable for the developers. Some condition they should add to their next sprint, for example. An error is something actionable that needs to be responded to soon. You probably don't need to send a page, but an error is a serious condition. Stack traces are extraordinarily useful in your logs. When code breaks (and it will), you want as much information surrounding that event as possible.
moe 14y ago
roll on an event (new day, process bounced etc.) not on reaching a 10Mb limit Only applicable when you produce a small, predictable volume of logging. Not a good general guideline because often you don't know how big your system may grow in the future. When logging high volumes then fixed chunk sizes are usually preferable (e.g. so you know how many you can keep around without a disk running full). Large sites produce multiple gigabytes of logs per hour. Use a header in the logfile Disagree. Context should be provided directly in each individual line. Logs get sliced and diced in analysis, fetching an arbitrary header from the right file for a given context can be rather painful. Don't be afraid of long lines. Use field delimited formats (TSV) so they can be easily parsed (either via 'cut' or a small script). stack traces ... should not be in the main logfile Wrong. When a serious error occurs then as much context as feasible must be provided. Stack traces are usually very feasible and indispensable.
- 3amOpsGuy 14y ago
  >> Only applicable when you produce a small, predictable volume of logging. No, It's primarily in the larger log volume systems we run into this problem. Take a FIX session adapter (very common software in the financial services world), baseline is around 5Gb/hour but its susceptible to fairly extreme fluctuation. When you roll the logs at say 500Mb, there's the chance our monitoring system misses an error in the tail of the previous log as the logs roll (which could in the worst cases mean loosing a couple of million or even more). Multiple logfiles for a day are harder to search through when handling the most common use case for log files, which is not debugging but ops staff answering questions on 'what happened with X?' What benefits does a split logfile provide? The one use case is transferring logs across the network. However most log management systems already split large logs up before transferring to the archive silo. You're loosing a lot, but gaining nothing with split logs. >> When logging high volumes It makes no difference, when the market swings you're going to need an extra 200Gb anyway, whether the logs are in chunks or not. In large volume apps deleting to make space is often not an option - you're logging in high volumes for a reason (often regulatory). >> fetching an arbitrary header from the right file for a given context can be rather painful. That's been a solved problem for a very long time. Just as recombining a separate trace log with a main app log based on time stamp is a solved problem. I believe this highlights the bigger problem, you can go to university or a.n. other school and learn to be a developer. Where can you go to learn operations? The net result is people come up with the own solutions each time instead of building on work already done. That work is available but its often passed on by word of mouth. Surprisingly hard even in today's world to find decent blogs on ops work (thankfully there are more around now). >> When a serious error occurs then as much context as feasible must be provided Yes this is the extraordinarily rare case where stack traces can be useful. They still don't belong in the main app log though. Push to a separate trace log and recombine on the fly when you need to view. The benefits are to the ops team and the monitoring system. Another approach not often mentioned is black box logging. Log to a circular buffer in memory, the first bytes of the buffer are set to a pattern that can be searched for in a core dump, the remainder is used as a circular buffer. It's fast to log to, takes no disk space and provides crucial context in the event of a crash. Im not sure I'll be able to convince you based on your tone, but speaking from experience, you are making the most common mistakes.

nitrogen 14y ago

Lastly stack traces, if they even have a use case at all in production, should not be in the main logfile. I would have to disagree with this one. There's no way to generate the stack trace after the fact if a particular bug is extremely rare; having a complete call chain makes fixing the problem much easier. Someone once said that removing error handling once the code is working is like removing the landing gear once a plane is airborne.

Pfhreak 14y ago

I always use warning to indicate something actionable for the developers. Some condition they should add to their next sprint, for example. An error is something actionable that needs to be responded to soon. You probably don't need to send a page, but an error is a serious condition. Stack traces are extraordinarily useful in your logs. When code breaks (and it will), you want as much information surrounding that event as possible.

moe 14y ago

roll on an event (new day, process bounced etc.) not on reaching a 10Mb limit Only applicable when you produce a small, predictable volume of logging. Not a good general guideline because often you don't know how big your system may grow in the future. When logging high volumes then fixed chunk sizes are usually preferable (e.g. so you know how many you can keep around without a disk running full). Large sites produce multiple gigabytes of logs per hour. Use a header in the logfile Disagree. Context should be provided directly in each individual line. Logs get sliced and diced in analysis, fetching an arbitrary header from the right file for a given context can be rather painful. Don't be afraid of long lines. Use field delimited formats (TSV) so they can be easily parsed (either via 'cut' or a small script). stack traces ... should not be in the main logfile Wrong. When a serious error occurs then as much context as feasible must be provided. Stack traces are usually very feasible and indispensable.

3amOpsGuy 14y ago
>> Only applicable when you produce a small, predictable volume of logging. No, It's primarily in the larger log volume systems we run into this problem. Take a FIX session adapter (very common software in the financial services world), baseline is around 5Gb/hour but its susceptible to fairly extreme fluctuation. When you roll the logs at say 500Mb, there's the chance our monitoring system misses an error in the tail of the previous log as the logs roll (which could in the worst cases mean loosing a couple of million or even more). Multiple logfiles for a day are harder to search through when handling the most common use case for log files, which is not debugging but ops staff answering questions on 'what happened with X?' What benefits does a split logfile provide? The one use case is transferring logs across the network. However most log management systems already split large logs up before transferring to the archive silo. You're loosing a lot, but gaining nothing with split logs. >> When logging high volumes It makes no difference, when the market swings you're going to need an extra 200Gb anyway, whether the logs are in chunks or not. In large volume apps deleting to make space is often not an option - you're logging in high volumes for a reason (often regulatory). >> fetching an arbitrary header from the right file for a given context can be rather painful. That's been a solved problem for a very long time. Just as recombining a separate trace log with a main app log based on time stamp is a solved problem. I believe this highlights the bigger problem, you can go to university or a.n. other school and learn to be a developer. Where can you go to learn operations? The net result is people come up with the own solutions each time instead of building on work already done. That work is available but its often passed on by word of mouth. Surprisingly hard even in today's world to find decent blogs on ops work (thankfully there are more around now). >> When a serious error occurs then as much context as feasible must be provided Yes this is the extraordinarily rare case where stack traces can be useful. They still don't belong in the main app log though. Push to a separate trace log and recombine on the fly when you need to view. The benefits are to the ops team and the monitoring system. Another approach not often mentioned is black box logging. Log to a circular buffer in memory, the first bytes of the buffer are set to a pattern that can be searched for in a core dump, the remainder is used as a circular buffer. It's fast to log to, takes no disk space and provides crucial context in the event of a crash. Im not sure I'll be able to convince you based on your tone, but speaking from experience, you are making the most common mistakes.