7 ms·

The most crucial thing that I've seen over the years is that most developers are simply afraid of bringing the application down on bugs. They conflate error ha

• bluepizza 2y ago
  I think the issue is that bringing the application down might mean cutting short concurrent ongoing requests, especially requests that will result in data mutation of some sort. Otherwise, some situations simply don't warrant a full shutdown, and it might be okay to run the application in degraded mode.
  • samiv 2y ago
    "I think the issue is that bringing the application down might mean cutting short concurrent ongoing requests, especially requests that will result in data mutation of some sort." Yes but what is worse is silently corrupting the data or the state because of running in buggy state.
    • jayd16 2y ago
      This is a false choice.
      • int_19h 2y ago
        If you don't know why a thing that's supposed to never be null ended up being null, you don't know what the state of your app is. If you don't know what the state of your app is, how do you prevent data corruption or logical errors in further execution?
        • victorbjorklund 2y ago
          You can "drop" that request which fails instead of crashing the whole app (and dropping all other requests too).
          • kstrauser 2y ago
            Sure. You wouldn't want a webserver to crash if someone sends a malformed request. I'd have to think long and hard about each individual case of running in degraded mode though. Sometimes that's appropriate: an OS kernel should keep going if someone unplugs a keyboard. Other times it's not: it may be better for a database to fail than to return the wrong set of rows because of a storage error.
          • buttercraft 2y ago
            That's exactly what the attacker wants you to do after their exploit runs: ignore the warning signs.
            • ndriscoll 2y ago
              You don't ignore it. You track errors. What you don't do is crash the server for all users, giving an attacker an easy way to DoS you.
              • 2y ago
• ramses0 2y ago
  "MIT v. Berkeley - Worse is Better" => https://blog.codinghorror.com/worse-is-better/ https://blog.codinghorror.com/worse-is-better/ "Fail Fast / Let it Crash" => https://erlang.org/pipermail/erlang-questions/2003-March/007870.html https://erlang.org/pipermail/erlang-questions/2003-March/007... ...you're in good company. :-)
• jayd16 2y ago
  Like everything in life, it depends. If this is some inconsequential part of the codebase it might be better to limp on then to completely stop anyone, user or fellow dev, from running the app at all. Said another way, graceful degradation is a thing.
  • samiv 2y ago
    How do you gracefully degrade when your program is in a buggy state and you no longer know what data is valid, what is garbage and what conditions hold ? If I told you to write a function that takes a chunk of customer JSON data but I told you that the data was produced / processed by some code that is buggy and it might have corrupted the data and your job is to write a function that works on that data how would you do it? Now your answer is likely to be "just check sum it", but what if i told you that the functions that compute the check sums sometimes go off rails in buggy branches and produce incorrect checksums. Then what? In a sane world your software is always well defined state. This means buggy conditions cannot be let to execute. If you don't honor this you have no chance of correct program.
    • gf000 2y ago
      Contrary to people's dislike of OOP, I think it pretty well solves the problem. You have objects, and calling a method on it may fail with an exception. If the method throws an exception, it itself is responsible for leaving behind a sane state, but due to encapsulation it is a feasible task. (Of course global state may still end up in illegal states, but if the program architecture is carefully designed and implemented it can be largely mitigated)
    • ndriscoll 2y ago
      Why not bring down the entire server if you detect an error condition in your application? You build things in a way where a job or request has isolated resources, and if you detect an error, you abort the job and free those resources, but continue processing other jobs. Operating systems do this through processes with different memory maps. Applications can do it through things like arenas or garbage collection.
• texuf 2y ago
  If you're validating parameters that originate from your program (messages, user input, events, etc), ASSERT and ASSERT often. If you're handling parameters that originate from somewhere else (response from server, request from client, loading a file, etc) - you model every possible version of the data and handle all valid and invalid states. Why? When you or your coworkers are adding code, the stricter you make your code, the fewer permutations you have to test, the fewer bugs you will have. But, you can't enforce an invariant on a data source that you don't control.
  • samiv 2y ago
    Yes of course the key here is to understand the difference between BUGS and logical (error) conditions. If I write an image processing application failing to process an image .png when: - user doesn't permission to the file - file is actually not a file - file is actually not an image - file contains a corrupt image etc. are all logical conditions that the application needs to be able to handle. The difference is that from the software correctness perspective none of these are errors. In the software they're just logical conditions and they are only errors to the USER. BUGS are errors in the software. (People often get confused because the term "error" without more context doesn't adequately distinguish between an error condition experienced by the user when using the software and errors in the program itself.)
  • keybored 2y ago
    > But, you can't enforce an invariant on a data source that you don't control. This is obvious.
• kstrauser 2y ago
  I largely agree. If it came to pass that the precondition fails, there's a bug somewhere and this code just hides it. At the very least, that should go to an error log that someone actually sees. I'm writing a Rust project right now where I deliberately put almost no error handling in the core of the code apart from the bits accepting user input. In Rust speak, I use .unwrap() all over the place when fetching a mandatory row from the DB or loading config files or opening a network connection to listen on or writing to stdout. If any of those things fail, there's not a thing I can plausibly do to recover from it in this context. I suppose I could write code like if let Ok(cfg) = load_config() { println!("Loaded the config without failing!"); Ok(cfg); } else { eprintln!("Oh no! Couldn't load the config file!"; Err("Couldn't load the config file"); } and make the program exit if it returns an error, but that's just adding noise around: return load_config().unwrap(); The only advantage is that the error message is more gentle, at the expense of adding a bunch of code and potentially hiding the underlying error message from the user so that they could fix it. I think Python also gets that right, where it's common to raise exceptions when exceptional things happen, and only ever handle the exceptions you can actually do something about. In 99.999% of projects, what are you actually going to do at the application level to properly deal with an OOM or disk full error? Nothing. It's almost always better to just crash and let the OS / daemon manager / top level event loop log that something bad happened and schedule a retry.
  • samiv 2y ago
    The whole story is 3-fold. We have - errors in the software itself, aka BUGS - logical conditions that are expected part of the program execution flow and expected state. some of these might be error conditions but only for the *user*. In other words they're not errors in the software itself. - unexpected failures where none of the above applies. typically only when some OS resource allocation fails, failed to allocate memory, socket, mutex etc and the reason is not because the programmer called the API wrong. In the first category we're dealing with BUGS and when I advocate asserting and terminating the process that only really applies to BUG conditions. If you let an application to continue in a buggy state then you cannot logically reason about it anymore. The logical conditions are the typical cases for example "file not found" or whatever. User tries to use the software but there's a problem. The application needs to deal with these but from the software correctness perspective there's no error. The error is only what the user perceives. When your browser prints "404" or "no internet connection" the software works correctly. The error is only from the user perspective. Finally the last category are those unexpected situations where something that should not fails. It is quite tricky to get these right. Is straight up exiting the right choice? Maybe the system will have more sources later if you just back off and try again later. Personally in C++ projects my strategy is to employ exceptions and let the callstack unwind to the UI level, inform the user and and then just return to the event loop. Of course the real trick is to keep the program state such that it's in some well defined state and not in a BUGGY state ;-)
• dehrmann 2y ago
  This is often called offensive programming.
  • samiv 2y ago
    Hot damn, I never heard of this term before but yeah that's exactly what it is. TIL, thanks.
    • layer8 2y ago
      Paradoxically, it is still a subset of defensive programming.
      • MathMonkeyMan 2y ago
        the best subset
• crabbone 2y ago
  I don't agree with any of this. First of all, this results in unintelligible errors. Linux is famous for abysmal error reporting, where no matter what the problem really is, you get something like ENOENT, no context, no explanation. Errors need to propagate upwards and allow the handling code to reinterpret them in the context of the work it was doing. Otherwise, for the user they are either meaningless or dangerous. Secondly, any particular function that encounters an unexpected condition doesn't have a "moral right" to terminate the entire program (who knows how many layers there are on top of what this particular function does?) Perhaps the fact that a function cannot handle a particular condition is entirely expected, and the level above this function is completely prepared to deal with the problem: insufficient permissions to access the file -- ask user to elevate permission; configuration file is missing in ~/.config? -- perhaps it's in /etc/? cannot navigate to URL? -- perhaps the user needs to connect to Wi-Fi network? And so on. What I do see in practice, is that programmers are usually incapable of describing errors in a useful way, and are very reluctant to write code that automates error recovery, even if it's entirely within reach. I think, the reason for this is that the acceptance criteria for code usually emphasizes the "good path", and because usually multiple bad things can happen down the "bad path", it becomes cumbersome and tiresome to describe and respond to the bad things, and then it's seldom done.
  • dvektor 2y ago
    yup. we have definitely all gotten an ENOENT or EIO before with no context.
• rikthevik 2y ago
  > The code is allowed to continue after it has logically bugged out. I'm a big fan of asserting preconditions and making it clear that we are getting into a bad place. I would rather dig through Sentry for an AssertionError than propagate a bad state and having to fix mangled data after the fact. If the AssertionError means that we mishandled valid user input, no problem, we'll go fix it. A few times in my career I've had to ask, "okay, how long has this bug been quietly mangling user data?" and it's not a fun place to be. Side note: I've never understood the convention of removing asserts in production builds. It seems like removing the seatbelts from the car before the race just to save a few pounds.
  • zarzavat 2y ago
    > Side note: I've never understood the convention of removing asserts in production builds. It seems like removing the seatbelts from the car before the race just to save a few pounds. Once an upon a time computers were slow and every cycle mattered. Assertions were compiled out of the build by necessity. Better a crash once in a while than the program hardly running because it was so slow.
    • drzaiusx11 2y ago
      Case in point: When I was stuck inside a "big ball of perl" codebase that heavily used assertions for method input validation, I generated a flame graph of where time was spent in the codebase and it turned out it was assertions all the way down. Since only a small percentage of inputs came from external/unvalidated sources (user input etc) it was fine to remove the vast majority of them outside of the development environment. So we turned them into no-ops in prod and had a significant performance improvement.
      • zbentley 2y ago
        Wouldn’t happen to have been a big healthcare company in Boston whose assert-anything function to validate that functions were called with correct signatures was called AssertFields?
• monlockandkey 2y ago
  Asserts are only available in debug compile mode.
• adonovan 2y ago
  I’m a big fan of assertion and rigorous preconditions but there are times when a failure of some invariant in a minor subsystem should not be allowed to crash the entire process, especially if the context makes it easy to return an error. In our project (the language server for Go) we have gotten tremendous value from telemetry: return an error, but report home the 1-bit fact that the assertion has failed. Often that fact is enough to figure out why; other times it is necessary to refine the assertion into two or more (in a later release) to get another bit or two of information about the nature of the failure.
• j1elo 2y ago
  Please, no! I worked with a 3rd party library that had this mentality. "A bug is a bug so the assert fails and thus the code is now in an unknown state thus The right thing to do would be to ABORT the process execution immediately". Oh my. Just do "if (pointer)" and when that fails, error out from the smallest context possible that applies to that pointer, and nothing more than that. I.e. the real BEST thing to do is to abort the current connection. To skip the current file with an error. To fail writing that piece of memory. Whatever. But never abort (unless maybe in debug builds). The end result of this library was that we had a WebRTC server handling 100s of simultaneous video calls, and then when a single new user tripped up during connection and went through a bogus code path, the library would decide "oh something is not as I expected so I'll abort, of course!" and the whole production server was brought down with it. That kind of behavior does not help achieving high production quality and providing robust and reliable services. We ended up removing the library's runtime assertions, which meant that connections that would bug the library code would just end up failing with an error somewhere else, that could be used to just discard the attempt and try again. All in all, numbers showed it was a huge positive in stability for the service.
  • ratmeadow 2y ago
    I think the original comment was more directed to scenarios where the precondition fail is let slide, NOTHING happens, and that's not the desired behaviour, so you have a bug. Such code exists too often and it's just poor quality. If there's anything I've learnt about error handling, it's that it must be approached with very careful consideration of what you want the app to do (conceptually) when the error occurs. Sometimes that's crash the program, sometimes it's throw an error, sometimes it's log and move on. The issue comes when devs don't want to think about this, whereby the simplest solution is absorb any error and forget about it.
    • mikeschinkel 2y ago
      Agree with your assumption, but then I also know that developers often read recommendations and convert them into dogma, ignoring the specifics for when those recommendations should be applied. Early in my career I taught programmers and I was horrified to come back to one client and find they followed my advice, but not where it applied meaning that what they did was actively harmful. Ever since that, I realized when a developer makes a recommendation they need to be very explicit about when it applies otherwise they are doing more harm than good and hopefully someone will publicly challenge their recommendation to bring their unstated qualifications to light.
• jminter 2y ago
  Absolutely. An intermediate path in Go is to recover any panics on your goroutines: in this case a nil dereference panic may cause the death of the goroutine but not the whole application. An example where this can be useful is in HTTP request handling: a single request might fail but the others can keep going -- but there are plenty of other use cases too. The panic recovery code can log for further investigation, as well as in the HTTP case for example probably returning a 500 to the caller if wanted. There are of course plenty of valid reasons not to take an approach like this too, but in some circumstances it can be useful.