3 ms·

Getting loads of core services out into third parties is really wonderful for logging. E.g. if email sending happens in Mandrill, then you never need to write d

by edmack 12y ago

Getting loads of core services out into third parties is really wonderful for logging. E.g. if email sending happens in Mandrill, then you never need to write decent logging calls for that and you have a reliable source of truth!

tlack 12y ago
Except you won't know if your server ever sent it to Mandrill. :) Always be extremely verbose with logging!
- reymus 12y ago
  I have always heard the opposite, that too much logging is as bad as no logging. I see the point of having the logs to be able to find out what happened, but what happens when there' s so much logging that the information needed is just buried into huge amount of noise?
  - msielski 12y ago
    This is true, without the right tools. I am moving to logstash with kibana to do this, and it's looking very promising. See http://www.elasticsearch.org/videos/kibana-logstash/ http://www.elasticsearch.org/videos/kibana-logstash/
    - revo13 12y ago
      Concur with this. Log everything, and use Logstash/Kibana to sift through it for what you are looking for.
  - gizmo686 12y ago
    My favorite systems to work with are the ones with overly verbose logs, where the overly verbose parts were clearly tagged and could be filtered out. Generally, we would never look at the verbose lines, and even when we did, we would normally have some idea what we were looking for, and be able to filter somewhat for it.
  - exelius 12y ago
    This was true before Splunk. If you logged too much, your logs could start to outstrip the assumptions behind your log rotations and cause trouble. Now the common wisdom is to just log everything so you can Splunk it later if you have a problem. Verbose logging + Splunk have made production incident identification so much easier than it used to be. Splunk DOES charge by the GB, but it's not very expensive in the long run.
- porker 12y ago
  This brings up a tangential problem I've yet to solve: how do you warn that something didn't happen when it should? E.g. you have a script that does backups. You log the script's output, but one day something fails and the script is no longer executed. Some form of dead man's handle is needed; the only way I can think of is to set up a monitoring service to check your log store for these entries every X hours. Any alternatives?
  - rwbhn 12y ago
    Monitor that the newest backup is less than N hours old.
    - caw 12y ago
      Alternatively, if you bother to have proper exit codes for your backup scripts, you could verify exit != 0, and alert on that. I have cron on all of my systems email STDERR to me, STDOUT is redirected to the normal logs. One you clean up your crons, you really only get email once a month or so when something breaks.
      - porker 12y ago
        Do you use something like http://habilis.net/cronic/ http://habilis.net/cronic/ ?
        caw 12y ago
        Nope, just instead of this 0 1 * * * backup >/var/log/foo 2>&1 I do this 0 1 * * * backup >/var/log/foo My errors are emailed to me via standard cron conventions. Normal logging is available for me if I care to look at it.
  - tlack 12y ago
    I've had this same issue over and over again in my career. I've toyed with the idea of writing a daily "sanity checker" in crontab that verifies various concepts of system health. Examples: Did the latest batch of data transfer to S3? Did we delete old customer accounts today? Did we get any signups (because if not, something may be broken, but not triggering an exception report etc)? Did we send out daily report emails? But I could see this easily becoming a pointless exercise, and I doubt I'd have the time to keep the sanity checker updated with the latest requirements. In fact, the sanity checker would probably become insane pretty quickly. Perhaps the platform itself should do this for you, in some way. Idea: while coding, indicate that this procedure should be running periodically, ie: Monitor.registerPeriodicTask('email-reports', 'daily') and then the system would log every time it occurs, with a generic task that would run periodically and scan for things that should have occurred, but haven't in some while.

by edmack 12y ago

tlack 12y ago

Except you won't know if your server ever sent it to Mandrill. :) Always be extremely verbose with logging!

reymus 12y ago
I have always heard the opposite, that too much logging is as bad as no logging. I see the point of having the logs to be able to find out what happened, but what happens when there' s so much logging that the information needed is just buried into huge amount of noise?
- msielski 12y ago
  This is true, without the right tools. I am moving to logstash with kibana to do this, and it's looking very promising. See http://www.elasticsearch.org/videos/kibana-logstash/ http://www.elasticsearch.org/videos/kibana-logstash/
  - revo13 12y ago
    Concur with this. Log everything, and use Logstash/Kibana to sift through it for what you are looking for.
- gizmo686 12y ago
  My favorite systems to work with are the ones with overly verbose logs, where the overly verbose parts were clearly tagged and could be filtered out. Generally, we would never look at the verbose lines, and even when we did, we would normally have some idea what we were looking for, and be able to filter somewhat for it.
- exelius 12y ago
  This was true before Splunk. If you logged too much, your logs could start to outstrip the assumptions behind your log rotations and cause trouble. Now the common wisdom is to just log everything so you can Splunk it later if you have a problem. Verbose logging + Splunk have made production incident identification so much easier than it used to be. Splunk DOES charge by the GB, but it's not very expensive in the long run.
porker 12y ago
This brings up a tangential problem I've yet to solve: how do you warn that something didn't happen when it should? E.g. you have a script that does backups. You log the script's output, but one day something fails and the script is no longer executed. Some form of dead man's handle is needed; the only way I can think of is to set up a monitoring service to check your log store for these entries every X hours. Any alternatives?
- rwbhn 12y ago
  Monitor that the newest backup is less than N hours old.
  - caw 12y ago
    Alternatively, if you bother to have proper exit codes for your backup scripts, you could verify exit != 0, and alert on that. I have cron on all of my systems email STDERR to me, STDOUT is redirected to the normal logs. One you clean up your crons, you really only get email once a month or so when something breaks.
    - porker 12y ago
      Do you use something like http://habilis.net/cronic/ http://habilis.net/cronic/ ?
      - caw 12y ago
        Nope, just instead of this 0 1 * * * backup >/var/log/foo 2>&1 I do this 0 1 * * * backup >/var/log/foo My errors are emailed to me via standard cron conventions. Normal logging is available for me if I care to look at it.
- tlack 12y ago
  I've had this same issue over and over again in my career. I've toyed with the idea of writing a daily "sanity checker" in crontab that verifies various concepts of system health. Examples: Did the latest batch of data transfer to S3? Did we delete old customer accounts today? Did we get any signups (because if not, something may be broken, but not triggering an exception report etc)? Did we send out daily report emails? But I could see this easily becoming a pointless exercise, and I doubt I'd have the time to keep the sanity checker updated with the latest requirements. In fact, the sanity checker would probably become insane pretty quickly. Perhaps the platform itself should do this for you, in some way. Idea: while coding, indicate that this procedure should be running periodically, ie: Monitor.registerPeriodicTask('email-reports', 'daily') and then the system would log every time it occurs, with a generic task that would run periodically and scan for things that should have occurred, but haven't in some while.