11 ms·

Log level 'error' should mean that something needs to be fixed

• shadowgovt 11mo ago
  This is the standard I use as well. In general, my rule of thumb is that if something is logging error, it would have been perfectly reasonable for the program to respond by crashing, and the only reason it didn't is that it's executing in some kind of larger context that wants to stay up in the event of the failure of an individual component (like one handler suffering a query that hangs it and having to be terminated by its monitoring program in a program with multiple threads serving web requests). In contrast, something like an ill-formed web query from an untrusted source isn't even an error because you can't force untrusted sources to send you correctly formed input. Warning, in contrast, is what I use for a condition that the developer predicted and handled but probably indicates the larger context is bad, like "this query arrived from a trusted source but had a configuration so invalid we had to drop it on the floor, or we assumed a default that allowed us to resolve the query but that was a massive assumption and you really should change the source data to be explicit." Warning is also where I put things like "a trusted source is calling a deprecated API, and the deprecation notification has been up long enough that they really should know better by now." Where all of this matters is process. Errors trigger pages. Warnings get bundled up into a daily report that on-call is responsible for following up on, sometimes by filing tickets to correct trusted sources and sometimes by reaching out to owners of trusted sources and saying "Hey, let's synchronize on your team's plan to stop using that API we declared is going away 9 months ago."
  • raldi 11mo ago
    Right. If staging or the canary is logging errors, you block/abort the deploy. If it’s logging warnings, that’s normal.
    • lanstin 11mo ago
      Unless it is logging more warnings because your new code is failing somehow; maybe it stopped parsing the reply correctly from a "is this request rate limited" service so it is only returning 429 to callers never accepting work.
  • nlawalker 11mo ago
    It seems that the easier rule of thumb, then, is that "application logic should never log an error on its own behalf unless it terminates immediately after", and that error-level log entries should only ever be generated from a higher-level context by something else that's monitoring for problems that the application code itself didn't anticipate.
• raldi 11mo ago
  Yes. Examples of non-defects that should not be in the ERROR loglevel: * Database timeout (the database is owned by a separate oncall rotation that has alerts when this happens) * ISE in downstream service (return HTTP 5xx and increment a metric but don’t emit an error log) * Network error * Downstream service overloaded * Invalid request Basically, when you make a request to another service and get back a status code, your handler should look like: logfunc = logger.error if 400 <= status <= 499 and status != 429 else logger.warning (Unless you have an SLO with the service about how often you’re allowed to hit it and they only send 429 when you’re over, which is how it’s supposed to work but sadly rare.)
  • jonathrg 11mo ago
    4xx is for invalid requests. You wouldn't log a 404 as an error
    • raldi 11mo ago
      I’m talking about codes you receive from services you call out to.
      • jonathrg 11mo ago
        Oh that makes sense.
        • raldi 11mo ago
          There are still some special cases, because 404 is used for both “There’s no endpoint with that name” and “There’s no record with the ID you tried to look up.”
          • deleted 11mo ago
            [deleted]
      • mewpmewp2 11mo ago
        What if user sends some sort of auth token or other type of data that you yourself can't validate and third party gives you 4xx for it? You won't know ahead of time whether that token or data is valid, only after making a request to the third party.
• jillesvangurp 11mo ago
  Errors mean I get alerted. Zero tolerance on that from my side.
• mfuzzey 11mo ago
  I think it's difficult to say without knowing how the system is deployed and administered. "If a SMTP mailer trying to send email to somewhere logs 'cannot contact port 25 on <remote host>', that is not an error in the local system" Maybe or maybe not. If the connection problem is really due to the remote host then that's not the problem of the sender. But maybe the local network interface is down, maybe there's a local firewall rule blocking it,... If you know the deployment scenario then you can make reasonable decisions on logging levels but quite often code is generic and can be deployed in multiple configurations so that's hard to do
  • greatgib 11mo ago
    The point is that if your program itself take note of the error from the library it is ok. You, as the program owner, can decide what to do with it (error log or not). But if you are the SMTP library and that you unilaterally log that as an error. That is an issue.
    • zamadatix 11mo ago
      The counterpoint made above is while what you describe is indeed the way the author likes to see it that doesn't explain why "an error is something which failed that the program was unable to fix automatically" is supposed to be any less valid a way to see it. I.e. should error be defined as "the program was unable to complete the task you told it to do" or only "things which could have worked but you need to explicitly change something locally". I don't even know how to say whether these definitions are right or wrong, it's just whatever you feel like it should be. The important thing is what your program logs should be documented somewhere, the next most important thing is that your log levels are self consistent and follow some sort of logic, and that I would have done it exactly the same is not really important. At the end of the day, this is just bikeshedding about how to collapse ultra specific alerting levels into a few generic ones. E.g. RFC 5424 defines 8 separate log levels for syslog and, while that's not a ceiling by any means, it's easy to see how there's already not really going to be a universally agreed way to collapse even just these down to 4 categories.
      • 11mo ago
• theli0nheart 11mo ago
  I agree with this. Not everything that a library considers an error is an application error. If you log an error, something is absolutely wrong and requires attention. If you consider such a log as "possibly wrong", it should be a warning instead.
• hnbear 11mo ago
  I have been particularly irritated in the past where people use a lower log level and include the higher log level string in the message, especially where it's then parsed, filtered, and alerted on my monitoring. eg. log level WARN, message "This error is...", but it then trips an error in monitoring and pages out. Probably breaching multiple rules here around not parsing logs like that, etc. But it's cropped up so many times I get quite annoyed by it.
  • jonathrg 11mo ago
    Stuff like that is a good argument for using structured logging, but even if you are just parsing text logs, surely you can make the parser be a bit more specific when retrieving the log level.
  • dragonwriter 11mo ago
    > I have been particularly irritated in the past where people use a lower log level and include the higher log level string in the message, especially where it's then parsed, filtered, and alerted on my monitoring. If your parsing, filtering, and monitoring setup parses strings that happen to correspond to log level names in positions other than that of log levels as having the semantics of log levels, then that's a parsing/filtering error, not a logging error.
• vpribish 11mo ago
  I just started playing in the Erlang ecosystem and they have EIGHT levels of logging messages. it seems crazily over-specific, but they are the champions of robust systems. I could live with 4 Error - alert me now. Warning - examine these later, Info - important context for investigations. Debug - usually off in prod.
  • regularfry 11mo ago
    The eight levels in Erlang are inherited from syslog, rather than something specific to Erlang itself.
  • groundzeros2015 11mo ago
    The first one should be crashing.
  • emmelaich 11mo ago
    I need Notice (between Info and Warning), for important events such as start and shutdown, and successfully connecting to the database, and ready to start serving. These otherwise would be in Info; and enabling Info level produces a torrent of uninteresting muck.
• dnautics 11mo ago
  let's say you a bunch of database timeouts in a row. this might mean that nothing needs to be fixed. But also, the "thing that needs to be fixed" might be "the ethernet cable fell out the back of your server". How do you know?
  • raldi 11mo ago
    You have an alert on what users actually care about, like the overall success rate. When it goes off, you check the WARNING log and metric dashboard and see that requests are timing out.
    • ImPostingOnHN 11mo ago
      That is a lagging indicator. By the time you're alerted, you've already failed by letting users experience an issue.
      • raldi 11mo ago
        What alternative would you propose? Page the oncall whenever there's a single query timeout?
        • dnautics 11mo ago
          the alternative i propose is have deep understanding of your system before popping off with dumb one size fits all rules that don't make sense.
      • danaris 11mo ago
        Well, yes. If the cable falls out of the server (or there's a power outage, or a major DDoS attack, or whatever), your users are going to experience that before you are aware of it. Especially if it's in the middle of the night and you don't have an active night shift. Expecting arbitrary services to be able to deal with absolutely any kind of failure in such a way that users never notice is deeply unrealistic.
• makeitdouble 11mo ago
  > This assumes an error/warning/info/debug set of logging levels instead of something more fine grained, but that's how many things are these days. Does it ? Don't most stacks have an additional level of triaging logs to detect anomalies etc ? It can be your New relic/DataDog/Sentry or a self made filtering system, but nowadays I'd assume the base log levels are only a rough estimate of whether an single event has any chance of being problematic. I'd bet the author also has strong opinions about http error codes, and while I empathize, those ships have long sailed.
• rwmj 11mo ago
  And the second rule is make all your error messages actionable. By that I mean it should tell me what action to take to fix the error (even if that action means hard work, tell me what I have to do).
  • andoando 11mo ago
    Error: Possible race condition, rewrite codebase
    • morkalork 11mo ago
      I have written out-of-band sanity checks that have caught race conditions, the recommendation is more like "<Thing> that should be locked, isn't. Check what was merged and deployed in the last 24h, someone ducked it up"
  • hyperadvanced 11mo ago
    This is just plain wrong, I vehemently disagree. What happens if a payment fails on my API, and today that means I need to go through a 20-step process with this pay provider, my database, etc. to correct that. But what’s worse is if this error happens 11,000 times and I run a script to do my 20 step process 11,000 times, but it turns out the error was raised in error. Additionally, because the error was so explicit about how to fix it, I didn’t talk to anyone. And of course, the suggested fix was out of date because docs lag vs. production software. Now I have 11,000 pissed off customers because I was trying to be helpful.
  • throw3e98 11mo ago
    Maybe that makes sense for a single-machine application where you also control the hardware. But for a networked/distributed system, or software that runs on the user's hardware, the action might involve a decision tree, and a log line is a poor way to convey that. We use instrumentation, alerting and runbooks for that instead, with the runbooks linking into a hyperlinked set of articles. My 3D printer will try to walk you through basic fixes with pictures on the device's LCD panel, but for some errors it will display a QR code to their wiki which goes into a technical troubleshooting guide with complex instructions and tutorial videos.
• jayofdoom 11mo ago
  In OpenStack, we explicitly document what our log levels mean; I think this is valuable from both an Operator and Developer perspective. If you're a new developer, without a sense of what log levels are for, it's very prescriptive and helpful. For an operator, it sets expectations. https://docs.openstack.org/oslo.log/latest/user/guidelines.html#definition-of-log-levels https://docs.openstack.org/oslo.log/latest/user/guidelines.h... FWIW, "ERROR: An error has occurred and an administrator should research the event." (vs WARNING: Indicates that there might be a systemic issue; potential predictive failure notice.)
  • quectophoton 11mo ago
    Thank you, this (and jillesvangurp's comment) sounds way more reasonable than the article's suggestion. If I have a daily cron job that is copying files to a remote location (e.g. backups), and the _operation_ fails because for some reason the destination is not writable. Your suggestion would get me _both_ alerts, as I want; the article's suggestion would not alert me about the operation failing because, after all, it's not something happening in the local system, the local program is well configured, and it's "working as expected" because it doesn't need neither code nor configuration fixing.
    • __turbobrew__ 11mo ago
      Agreed, I don’t get the OPs delineation between local and non-local error sources. If your code has a job to do it doesn’t matter if the error was local or non-local, the operator needs to know that the code is not doing its job. In the case of something like you cannot backup files to a remote you can try to contact the humans who own the remote or come up with an alternative backup mechanism.
• HarHarVeryFunny 11mo ago
  I agree with the sentiment, although not sure if "error" is the right category/verbiage for actionable logs. In an ideal world things like logs and alarms (alerting product support staff) should certainly cleanly separate things that are just informative, useful for the developer, and things that require some human intervention. If you don't do this then it's like "the boy that cried wolf", and people will learn to ignore errors and alarms since you've trained them to understand that usually no action is needed. It's also useful to be able to grep though log files and distinguish failures of different categories, not just grep for specific failures.
• eterm 11mo ago
  How I'd personally like to treat them: - Critical / Fatal: Unrecoverable without human intervention, someone needs to get out of bed, now. - Error : Recoverable without human intervention, but not without data / state loss. Must be fixed asap. An assumption didn't hold. - Warning: Recoverable without intervention. Must have an issue created and prioritised. ( If business as usual, this could be downgrading to INFO. ) The main difference therefore between error and warning is, "We didn't think this could happen" vs "We thought this might happen". So for example, a failure to parse JSON might be an error if you're responsible for generating that serialisation, but might be a warning if you're not.
  • mewpmewp2 11mo ago
    What if you are integrated to a third party app and it gives you 5xx once? What do you log it as, and let's say after a retry it is fine.
    • bqmjjx0kac 11mo ago
      I would log a warning when an attempt fails, and an error when the final attempt fails.
      • mewpmewp2 11mo ago
        You are not the OP, but I think I was trying to point out this example case in relation to their descriptions of Error/Warnings. This scenario may or may not yield in data/state loss, it may also be something that you, yourself can't immediately fix. And if it's temporary, what is the point of creating an issue and prioritizing. I guess my point is that to any such categorization of errors or warnings there are way too many counter examples to be able to describe them like that. So I'd usually think that Errors are something that I would heuristically want to quickly react to and investigate (e.g. being paged, while Warnings are something I would periodically check in (e.g. weekly).
        • wredcoll 11mo ago
• teo_zero 11mo ago
  This doesn't resonate with my experience. I place the line between a warning and an error whether the operation can or can't be completed. A connection timed out, retrying in 30 secs? That's a warning. Gave up connecting after 5 failed attempts? Now that's an error. I don't care so much if the origin of the error is within the program, or the system, or the network. If I can't get what I'm asking for, it can't be a mere warning.
• AndroTux 11mo ago
  “cannot contact port 25 on <remote host>” may very well be a configuration error. How should the program know?
  • HankB99 11mo ago
    Would it make sense to consider anything that prevents a process from completing it's intended function an error? It seems like this message would fall into that category and, as you pointed out, could result from a local fault as well.
  • kijin 11mo ago
    SMTP clients are designed to try again with exponential backoff. If the final attempt fails and your email gets bounced, now that's an error. Until then, it's just a delay, business as usual.
    • deleted 11mo ago
      [deleted]
  • notatoad 11mo ago
    >How should the program know? if we're talking about logs from our own applications that we have written, the program should know because we can write it in a way that it knows. user-defined config should be verified before it is used. make a ping to port 25 to see if it works before you start using that config for actual operation. if it fails the verification step, that's not an error that needs to be logged.
    • 1718627440 11mo ago
      So when the random error on a remote party happens at one time your system ignores it, bu when it happens at another time, it prevents the server from booting? That's a very brittle system.
      • notatoad 11mo ago
        log level error prevents your server from booting? i'm pretty sure that's not how logging works.
• mkoubaa 11mo ago
  To me it's always a neat trick when you're not allowed to use print() in production code
• aqme28 11mo ago
  I agree with this take in a steady state, but the process of building software is just that-- it's a process. So it's natural for error messages to be expected, as you progressively add and then clear up edge cases.
  • raldi 11mo ago
    Exactly: When you're building software, it has lots of defects (and, thus, error logging). When it's mature, it should have few defects, and thus few error logs, and each one that remains is a bug that should be fixed.
    • plorkyeran 11mo ago
      I don't understand why you seem to think you're disagreeing with the article? If you're producing a lot of error logs because you have bugs that you need to fix then you aren't violating the rule that an error log should mean that something needs to be fixed.
      • raldi 11mo ago
        I couldn’t agree more with the article. What made you think I disagreed?
• tgv 11mo ago
  I log authorization errors as errors. Are they errors? It depends on how you read the logs. Perhaps you want to distinguish between internal, external and non-attributable errors for easier grepping.
• layer8 11mo ago
  > When implementing logging, it's important to distinguish between an error from the perspective of an individual operation and an error from the perspective of the overall program or system. Individual operations may well experience errors that are not error level log events for the overall program. You could say that an operation error is anything that prevents an operation from completing successfully, while a program level error is something that prevents the program as a whole from working right. This is a nontrivial problem when using properly modularized code and libraries that perform logging. They can’t tell whether their operational error is also a program-level error, which can depend on usage context, but they still want to log the operational error themselves, in order to provide the details that aren’t accessible to higher-level code. This lower-level logging has to choose some status. Should only “top-level” code ever log an error? That can make it difficult to identify the low-level root causes of a top-level failure. It also can hamper modularization, because it means you can’t repackage one program’s high-level code as a library for use by other programs, without somehow factoring out the logging code again.
  • ivan_gammel 11mo ago
    Libraries should not log on levels above DEBUG, period. If there’s something worthy for reporting on higher levels, pass this information to client code, either as an event, or as an exception or error code.
    • layer8 11mo ago
      From a code modularization point of view, there shouldn’t really be much of a difference between programs and libraries. A program is just a library with a different calling convention. I like to structure programs such that their actual functionality could be reused as a library in another program. This is difficult to reconcile with libraries only logging on a debug level.
      • ivan_gammel 11mo ago
        The main difference is that library is not aware of the context of the execution of the code, so cannot decide, whether the problem is expected, recoverable or severe.
• leni536 11mo ago
  I make error logs fail happy path functional/integration tests for the backend applications I'm currently writing.
• georgefrowny 11mo ago
  Easy to say, but there's "yes we know this is wrong but this will have to do for now" and "we don't expect to see this in real life unless something has gone sideways".
  • oofbey 11mo ago
    At scale the rare events start to happen reliably. Hardware failures almost certainly cause ERROR conditions. Network glitches. Our production system pages oncall for any errors. At night it will only wake somebody up for a whole bunch of errors. This discipline forces us to take a look at every ERROR and decide if it is spurious and out of our control or something we can deal with. At some point our production system will reach a scale where there are errors logged constantly and this strategy Durant make sense any more. But for now it helps keep our system clean.
    • georgefrowny 11mo ago
      I think if someone is going be gotten out of bed that would be a critical rather then error. Generally I'd say in a large "live" system, errors end up raising Jira tickets, criticals end up ringing phones.
      • oofbey 10mo ago
        Most systems I’ve worked with can go completely offline without ever logging a critical error. Some coding errors or misconfiguration or failure in a critical system - enough to log an error - and nobody can get any useful work done. I’ve never seen sobering that cash convert those into critical errors. I’m used to critical errors being rare - certain failures of a server to start. Or infra problems.
• mschuster91 11mo ago
  > If error level messages are not such a sign, I can assure you that most system administrators will soon come to ignore all messages from your program rather than try to sort out the mess, and any actual errors will be lost in the noise and never be noticed in advance of actual problems becoming obvious. Bold of you to assume that there are system administrators. All too often these days it's "devops" aka some devs you taught how to write k8s yamls.
• Too 11mo ago
  Agree with the post. The job of blackbox is to turn probes into metrics. If a probe fails, that should just become a probe_success=0 metric. Blackbox did its job and should not log an error.
• plandis 11mo ago
  I agree. Error or higher should result in an alarm and indicates that some corrective action needs to be taken.
• BiraIgnacio 11mo ago
  It means something is wrong, yes. Now, if it's worth fixing (granted, most of the time it would), that's another story.
• Kinrany 11mo ago
  Why are logs usually assumed to be for human consumption only? It seems weird to me that log storage usually exists outside of the system and isn't a general purpose message bus.
• azov 11mo ago
  If my system doesn’t work - I want to be alerted. If notification was supposed to be sent but wasn’t - it’s an error regardless of whether it wasn’t sent because of a bug in my code or external service being down. It may be a warning if I’m still retrying, but if I gave up - it’s an error. “External service down, not my problem, nothing I can do” is hardly ever the case - e.g. you may need to switch to a backup provider, initiate a support call, or at least try to figure out why it’s down and for how long.
• 29athrowaway 11mo ago
  Input errors do not need fixing, so no.
  • lanstin 11mo ago
    If they cause your customers to ditch your product but calling them and saying "your calls are all getting 4xx because you are not putting the state code into the call parameters" would keep them as customers, then you would be wise to make that communication.
    • dolmen 11mo ago
      But first ensure that the input error is properly reported to the client in the response body (ideally in a structured way), so the client could have figured out by himself. If a fix is needed on your side for this matter, having a conversation with a customer might be useful before breaking more stuff. ("We have no state code in EU. Why is that mandatory?").
      • lanstin 10mo ago
        If you are trying to sell a product, it is sometimes useful to solve people problems for them, rather than counting on them to figure them out on their own.
• hedayet 11mo ago
  I agree with the principle: log level error should mean someone needs to fix something. This post frames the problem almost entirely from a sysadmin-as-log-consumer perspective, and concludes that a correctly functioning system shouldn’t emit error logs at all. That only holds if sysadmins are the only "someone" who can act. In practice, if there is a human who needs to take action - whether that’s a developer fixing a bug, an infra issue, or coordinating with an external dependency - then it’s an error. The solution isn’t to downgrade severity, but to route and notify the right owner. Severity should encode actionability, not just system correctness.
• winningChild 11mo ago
  I have a collection of cameras, I can take a picture or photos with those cameras. Some of the lenses may not work properly with the lighting. That doesn’t mean the object being photographed is faulty.
• peanut-walrus 11mo ago
  Disagree. If you have an error that NEEDS fixing, your program should exit. Error level logs for operation level errors are fine.
• blkflcn3 11mo ago
  > What an error log level should mean (a system administrator's view) That says it all: - Backseat driving - Not a developer by trade
• dpc_01234 11mo ago
  Error log level should be renamed. It's just a terrible name that confuses usage.
  • fogzen 11mo ago
    Yeah, even alert/warn/info would be an improvement. I hate the concept of “errors” in general. They’re an excuse to avoid responsibility, and ship broken software with known undefined behavior. The very notion of an error basically means “there was behavior I chose to not handle and do anything about but which I knew would happen” which is essentially just negligence.
• jedberg 11mo ago
  I feel like it's more nuanced than OP writes. Presumably every log line comes from something like a try/catch. An edge case was identified, and the code did something differently. Did it do what it was supposed to do, but in a different way or defer for retrying later? Then WARN. Did it fail to do what it needed to do? ERROR Did it do what it needed to do in the normal way because it was totally recoverable? INFO Did data get destroyed in the process? FATAL It should be about what the result was, not who will fix it or how. Because that might change over time.
  • Joker_vD 11mo ago
    > Did it do what it was supposed to do, but in a different way or defer for retrying later? Then WARN. > Did it fail to do what it needed to do? ERROR > Did it do what it needed to do in the normal way because it was totally recoverable? INFO We have a web-facing system (it uses a custom request-response protocol on top of Websocket... it's an old system) that users are routinely trying to, ahem, automate even though it's technically against ToS but hey, as long as we don't catch them? Anyway, it's quite often to see user connections that send malformed commands and then get disconnected after we send them a critical_error/protocol_error message — we do have quite extensive validation logic for user commands. So, how should such errors be logged in your opinion? I know that we originally logged them as errors but very quickly changed to warnings, and precisely for the reasons outlined in TFA: if some kewl haxxor can't figure out how to quote strings in JSON, it's not really something we can't fix. We probably should keep the records, just to know that "oh, some script kiddie was trying to hack us during that time period" but nothing more than that; it definitely doesn't warrant the "hey, there are too many errors in sfo2 location, please take a look" summons at 3:00 AM from the ops team.
    • jedberg 11mo ago
      It sounds like it did exactly what it was supposed to do -- reject the bad input. Looks like an INFO to me.
• Waterluvian 11mo ago
  I think this is one of those discussions where there's no one right answer (though there's many wrong answers). All you have to do is pick a reasonable definition, write it down, socialize it, and be consistent when using it. I think discussions that argue over a specific approach are a form of playing checkers.
• rsanek 11mo ago
  If something needs to be fixed, why is it just a log? How is someone supposed to even notice a random error log? At the places that I've worked, trying to make alerting be triggered on only logs was always quite brittle, it's just not best practice. Throw an exception / exit the program if it's something that actually needs fixing!
  • Copenjin 11mo ago
    > If something needs to be fixed, why is it just a log? What he meant is that is an unexpected condition, that should have never happened, but that did, so it needs to be fixed. > How is someone supposed to even notice a random error log? Logs should be monitored. > At the places that I've worked, trying to make alerting be triggered on only logs was always quite brittle, it's just not best practice. Because the logs sucked. It not common practice, it should be best practice. > Throw an exception / exit the program if it's something that actually needs fixing! I understand the sentiment, but some programs cannot/should not exit. Or you have an error in a subsystem that should not bring down everything. I completely agree with the approach of the author, but also understand that good logging discipline is rare. I worked in many places where logs sucked, they just dumped stuff, and had to restructure them.
    • lanstin 11mo ago
      While it is fun to have your code run for 500 days without restart, it is a bad architecture. You should be able to move load around from host to host or network to network without losing any work. This involves graceful draining and then shutting down the old. For impossible errors exiting and sending the dev team as much info as possible (thread dump, memory dump, etc) is helpful. In my experience logs are good for finding out what is wrong once you know something is wrong. Also if the server is written to have enough but not too much logging you can read them over and get a feel for normal operation.
• mycall 11mo ago
  Severity is the value and you set thresholds based on context of the error type.
• umpalumpaaa 11mo ago
  What I like about objective-c’s error handling approach is that a method that can fail is able to tell if a caller considers error handling or not. If the passed *error is NULL you know that that is no way for a caller to properly handle the error. My implementations usually have this logic: if error == NULL and operationFailed then log error Otherwise Let client side do the error handling (in terms of logging)
• Glyptodon 11mo ago
  I agree errors should be errors. Many things that are logged for other reasons should use a different label. That said, the thing I've cone find being useful as a subcategory of error are errors due to data problems vs errors due to other issues.
• jmull 11mo ago
  I encourage people to think a few moments about what to log and at what level. You’re kind of telling a story to future potential trouble-shooters. When you don’t think about it at all (it doesn’t take much), you tend to log too much and too little and at the wrong level. But this article isn’t right either. Lower-level components typically don’t have the context to know whether a particular fault requires action or not. And since systems are complex, with many levels of abstractions and boxes things live in, actually not much is in a position to know this, even to a standard of “probably”.
• Xss3 11mo ago
  Some programs are error resistant and need an additional level: Fatal. A warning can be ignored safely. Warnings may be 'debugging enabled, results cannot be certified' or something similar. An error should not be ignored, an operation is failing, data loss may be occurring, etc. Some users may be okay with that data loss or failing operation. Maybe it isnt important to them. If the program continues and does not error in the parts that matter to the user, then they can ignore it, but it is still objectively an error occurring. A fatal message cannot be ignored, the system has crashed. Its the last thing you see before shutdown is attempted.
  • deleted 11mo ago
    [deleted]
• aunty_helen 11mo ago
  Good logging is critical and actually having the logs turned on in production. No point writing logs if you silence them. My company now has a log aggregator that scans the logs for errors, when it finds one, creates a Trello card, uses opus to fix the issue and then propose a PR against the card. These then get reviewed, finished if tweaks are necessary and merged if appropriate.
• Insanity 11mo ago
  Coincidentally was reviewing code yesterday that had a confusing/contradictory statement.. error_msg = "xyz went wrong" log.warn(error_msg) My comment on the CR was about this being an inherent contradiction and incredibly confusing to know if it's actually an error or a warning..
• bandrami 11mo ago
  It's like how an alert system that sends more than ~8 alerts a day effectively sends zero alerts.
• bytefish 11mo ago
  Making software is 20% actual development and 80% is maintenance. Your code and your libraries need to be easy to debug, and this means logs, logs, logs, logs and logs. The more the better. It makes your life easy in the long run. So the library you are using fires too many debug messages? You know, that you can always turn it off by ignoring specific sources, like ignoring namespaces? So what exactly do you lose? Right. Almost nothing. As for my code and libraries I always tend to do both, log the error and then throw an exception. So I am on the safe side both ways. If the consumer doesn’t log the exception, then at least my code does it. And I give them the chance to do logging their way and ignore mine. I am doing a best-guess for you… thinking to myself, what’s an error when I’d use the library myself. You don’t trust me? Log it the way you need to log it, my exception is going to transport all relevant data to you. This has saved me so many times, when getting bug reports by developers and customers alike. There are duplicate error logs? Simply turn my logging off and use your own. Problem solved. If it is a program level error, maybe a warning and returning the error is the correct way to do. Maybe it’s not? It depends on the context. And this basically is the answer to any software design question: It depends.
  • szundi 11mo ago
    [dead]
• knallfrosch 11mo ago
  > If a SMTP mailer trying to send email to somewhere logs 'cannot contact port 25 on <remote host>', that is not an error in the local system and should not be logged at level 'error'. A mail program not being to checks notes send emails sounds like an error to me. (Unless you implement retries.)
  • deleted 11mo ago
    [deleted]
• t43562 11mo ago
  Errors can be recovered automatically sometimes but at the level at which you log them you don't know if that's going to happen. I therefore think this suggestion is not easy to follow. Even if your libraries use nothing but exceptions or return codes you still end up with levels. You still end up with logs that have information in them that gets ignored when it shouldn't be because there's so much noise that people get tired of all the "cries of wolf." Occasionally one is at a high enough level to know for sure that something needs fixing and for this I use "CRITICAL" which is my code for "absolutely sure that you can't ignore this." IMO it's about time AI was looking at the logs to find out if there was something we really need to be alerted to.
• alex-moon 11mo ago
  "If a SMTP mailer trying to send email to somewhere logs 'cannot contact port 25 on <remote host>', that is not an error in the local system and should not be logged at level 'error'." But it is still an error condition, i.e. something does need to be fixed - either something about the connection string (i.e. in the local system) is wrong, or something in the other system or somewhere between the two is wrong (i.e. and therefore needs to be fixed). Either way, developers on this end (I mean someone reading the logs - true that it might not be the developers of the SMTP mailer) need to get involved, even if it is just to reach out to the third party and ask them to fix it on their end. A condition that fundamentally prevents a piece of software from working not being considered an error is mad to me.
  • jeroenhd 11mo ago
    Exactly this, a remote error may still be your problem. If your SMTP mailer is failing to send out messages on behalf of your customer because their partners' email servers cannot be reached, your customer is still going to ask you why the documents never arrived. Plus, a remote server not being reachable doesn't say anything about where the problem lies. Did you mess up a routing table? Did your internet connection get severed? Did you firewall off an important external server? Did you end up on a blacklist of some kind? These types of messages are important error messages for plenty of people. Just because your particular use case doesn't care about the potential causes behind the error doesn't mean nobody does.
  • jeremyjh 11mo ago
    There is no "connection string" in mail software that defines the remote host. The other party's MX records do that. If you are sending mail to thousands of remote hosts and one is unreachable, that is NOT a problem a mail administrator is going to be researching or trying to fix because they cannot, and it is not their problem. Either the email address is wrong, the remote host is down, or its DNS is misconfigured. This happens constantly all day long everywhere. The errors are reported to the sender of the email, which is the person who has the problem to solve.
    • alex-moon 11mo ago
• yoan9224 11mo ago
  I've found the most practical rule is: "Would I want to be paged for this at 2 AM?" If yes: ERROR If I want to check it tomorrow: WARNING If it's useful for debugging: INFO Everything else: DEBUG The problem with the article's approach is that libraries don't have enough context. A timeout calling an external API might be totally fine if you're retrying, but it's an ERROR if you've exhausted retries and failed the user's request. We solve this by having libraries emit structured events with severity hints, then the application layer decides the final log level based on business impact. A 500 from a recommendation service? Warning. A 500 from the payment processor? Error.
• deleted 11mo ago
  [deleted]
• twosdai 11mo ago
  I understand where the author is coming from but frankly I think this design pattern is not really correct. Obviously this depends on teams, application context and code bases. But "knowing if action needs to be taken" can't be boiled into a simple log level for most cases. There is a reason most alerting software like pagerduty is just a trigger interface and the logic for what constitutes the "error" is typically some data level query in something like datadog, sumologic, elastic search, or graphana, that either looks for specific string messages, error types, or a collection of metric conditions. Cool if you want to consider that any error level log needs to be an actionable error but what quickly happens is that some error cases are auto retry able due to infrastructure conditions that the application has completely no knowledge of. And to run some sort of infrastructure query at error write time in code, eg 1. Error is thrown 2. Prior to logging guess/determine if the case can be retired through a few http calls. 3. Log either a warning or an error Seems to be a complete waste when we could just write some sort of query in our log/metrics management platform of choice which takes into account the infrastructure conditions for us.