9 ms·

Assertions in Production Code? (2008)

100 points by ScottWRobinson 8y ago

Const-me 8y ago
In my code, most asserts become log messages in release builds. This helps a lot when troubleshooting problems, customers don't have debuggers installed. I often work on desktop software, unfortunately it's impossible to achieve avionics level of reliability because I don't have a luxury of controlled hardware and software environment.
gilgoomesh 8y ago
The article argues in favor of assertions in production code but acknowledges that the author's experience is from a very specific field with very specialized constraints and extensive design around recovering from aborted components. This advice is not obviously applicable to different kinds of programming with different constraints and different environments. As with so many things: one solution does not fit all problems.
- Nomentatus 8y ago
  You're gainsaying, but others have dropped empirical evidence on the other side, averring a low to no downside; and my experience is all on the other side too. It would help if you gave some lived examples.
sbradford26 8y ago
I think part of what is argued is that people need to look at their software and determine what failure modes are not acceptable. It is impossible to design a system that won't fail so the focus should be on designing parts that detect and handle failures. For those parts you may want to pull in techniques from aerospace such as command monitor setups, assertions, fail-safe modes, and force restarts. Clearly not all software is flight critical but a lot of software could be improved by treating some parts of it as such.
pmichalina 8y ago
Assertions, throws, and try-catch are code smells. Really bad design choice considering we have more elegant solutions like monads.
- TeMPOraL 8y ago
  Assertions, yes, in so far as they mean "I know this can't happen, but I can't structurally prevent it in the program in an elegant fashion, with the tools I use". However, I can't see how monads are suddenly going to make this problem disappear. As for throws/try-catch - beyond assertion use, they also report all problems that are outside your control (like IO failing due to HDD damage, or connection dropping). Those kinds of problems are here to stay.
- oftenwrong 8y ago
  Sure, monads provide an elegant way to structure code with error conditions, but I don't see a problem with using simple assertions, throws, try-catch to enforce invariants. Personally, I don't care about elegance as much as whether or not it works, as long as it is simple to use and maintain. I can return something like a Maybe/Either, or I can throw an exception - both approaches are fine by me if it forces the programmer to account for error cases.
  - pmichalina 8y ago
    Throwing is a side effect breaking flow of control for no good reason. It’s takes you down the same path as all other spaghetti code. Not as much elegance as it is about purity. Minimizing side effects is the number one way to reduce bugs. It should be the number one guiding principle when creating out reliable software. Which means, you simply cannot use the primitive try catch or similar construct. Don’t break flow of control. Guide it to a terminal value instead. These days, when I see try-catch and if-else constructs (which is in most codebases) it’s clear there will be bugs over the life of the application. It’s fine, use them, but there is a world of greatness when you ditch these faulty constructs. Just like ditching OOP constructs. All built on false premises.
    - 8y ago
dougk16 8y ago
There's a good approach to asserts I adopted years ago for your typical low-risk CRUD/web/mobile app that I'm guessing ~90% of us are working on (as opposed to aviation software like in the article): Use asserts as you normally would, but in production have it fire an event to google analytics or some such. This is particularly useful for those middle-ground situations where a full app crash isn't justified because things will probably chug along fine, but nonetheless you want to know something's screwy. For debug/testing mode I usually pop a toast or dialog so the tester knows to report something to me even if the app is otherwise working fine. You often say to yourself "Well I know this assert will never trigger but I'll put it here just in case". Then you check your analytics next day after release and get humbled.
- vram22 8y ago
  >You often say to yourself "Well I know this assert will never trigger but I'll put it here just in case". Then you check your analytics next day after release and get humbled. That is exactly what asserts are for: checking for things that "cannot happen" - but do sometimes happen.
roland35 8y ago
Software development often does seem like a struggle between reliability/robustness and safety/correctness. Figuring out what to do with asserts falls in this debate! For embedded development I have often had ASSERTs log out to a serial port with the function name/line # and continue operating - this can then be connected to any logging device. This makes the system a little more robust (it won't crash, right away at least) but less correct/safe. During development testing we often had complaints of machines freezing due to asserts, which always increased the priority of those bugs! Definitely a good thing in the long run to fix those bugs and make the system more correct. Stopping all executing may not be the correct 'safe state' for an airplane though!
- sbradford26 8y ago
  Usually the hardware is tri or even quad redundant so the system can handle one of the boxes halting execution while it is either reset or other action is taken. If several failures happen one box may be powered off by the others to stop erroneous outputs from the failed box.
  - WalterBright 8y ago
    May? The offending box gets electrically isolated from the craft if any fault is detected in it. Any fault. Now, the pilot may attempt to restart it by manipulating the circuit breakers, but that's the pilot's option.
- WalterBright 8y ago
  > Stopping all executing may not be the correct 'safe state' for an airplane though! It absolutely is. It's a (literally) disaster to allow code that has entered an unknown state to control aircraft functions. There is NO WAY such a design would EVER be certified by the FAA.
  - cryptonector 8y ago
    In a fly-by-wire design (which is de rigeur now), stopping all operations would be way worse than continuing.
alangpierce 8y ago
At my work (hosted enterprise webapp), both the frontend and backend have both an `assert` function and an `assertAndContinue` function. `assert` always crashes on failure, and `assertAndContinue` crashes in dev and logs an error and keeps going in prod. Each time we want to verify a runtime assumption, we decide which type of assert to use. We prefer `assertAndContinue` (and I push for it in code review), but there are reasons to use plain assert: • In some cases, crashing is a better user experience than proceeding in an unknown state. Usually this is on the backend when data integrity is at risk, but could be on the frontend when we're at risk of making a server call with the wrong arguments. But usually, pretty much anything that doesn't crash is better than crashing. • `assertAndContinue` requires that you have a reasonable fallback. If code is going to crash the next line anyway, there's no point in `assertAndContinue`. In most cases, it's easy, but sometimes it takes real engineering, like building in an error state into the UI component that skips downstream code. When there isn't an obvious fallback, using `assertAndContinue` is a judgement call based on the difficulty of implementing a real fallback, the likelihood of the bug actually happening, and severity if the bug does trigger.
- wvenable 8y ago
  I have a difficult time imagining anytime that assertAndContinue makes sense. If a fallback is necessary and recovery possible, I would have asserts throw and use normal exceptional handling for that.
  - MaulingMonkey 8y ago
    Exceptions are a control flow mechanism, not necessarily an error collection mechanism. I also frequently work with languages either without exception handling or with exception handling disabled for certain targets. Even when exception handling is enabled, it's a bit silly and heavyweight if you just need to short circuit a single conditional, loop, or function. Why throw when you can break or return? And when you do use nonfatal-but-logged exceptions, what do you put in the exception handler and what makes it different from assertAndContinue besides minor errata like being specialized for exceptions?
tedd4u 8y ago
I recently discussed this with an engineering manager for a notable consumer iOS app with tens of millions of downloads. In discussing how to better reduce their crash rate, there was a debate on his team about whether to enable assertions in their production iOS app. As any iOS developer knows, even if you have a great crash reporter and database there are a lot of crashes that are hard to figure out. One faction believed you should never intentionally crash the app, while another faction believed the app was going to crash anyways or maybe do something worse. They decided to test it out and during a sprint that was already mostly about bug fixes where they would do extra testing before release, they enabled asserts, and closely monitored their crash reporter. What they found was that the number of crashes did not change much, but the cardinality went down, significantly. The learning was code executing past a disabled assertion may be in one of n different bad states, each of which might lead to a different type of crash. They now had better high-level information about what was causing crashes (knowing which asserts were wrong) and it helped them reduce their crash rate much more than raw crashes without asserts (including cases where the crash was in iOS code, not app code).
- bsaul 8y ago
  That's a really interesting results. Those are the kind of thought you may have but knowing that it actually worked on a production app that's massively used is a really valuable thing. Anyone else had similar experience ?
  - noir_lord 8y ago
    I inherited a hilarious bad(ly written) legacy system. One of the first things I did was turn on every possible "throw on error, die on error, assert" in development (could't do it production as it'd just fall over constantly). The constant crashes in dev meant I rapidly fixed the low hanging fruit (undefined indexes etc) and then slowly as I cleaned an area up I'd leave the asserts in once in production since at that point what I'm asserting is "I think I fixed all the insanity but tell me straight away if my assumption was bad". Painful very painful but those areas now cause me very few if any issues.
  - pfranz 8y ago
js8 8y ago
I completely agree with the approach outlined in the original post. (It's also called defensive programming.) Not failing fast and hard when the application is in unexpected state (which is what assert should check) is robust only on the surface. I think we have the debate because it's not just about adding asserts. If you want to add asserts to existing codebase, essentially making it to fail fast and hard, you need to have a system architecture that is capable of dealing with said failure. If your codebase isn't architected as such, you will probably make things worse for the user by adding extra asserts. I think that's what the people who are against asserts fear of, and rightfully so. But the answer shouldn't be do not use asserts, the answer should be make the system more resilient to failure first.
dwohnitmok 8y ago
It doesn't fix everything, but judicious use of private data constructors and custom types can go a long way towards reducing the need for asserts. They can bubble up potential violations of preconditions and force handler code to be written. Some Scala examples, but they're hopefully quite language agnostic: https://m.youtube.com/watch?v=Csj3lzsr0_I https://m.youtube.com/watch?v=Csj3lzsr0_I https://m.youtube.com/watch?v=keTId618iOs https://m.youtube.com/watch?v=keTId618iOs
draw_down 8y ago
Soft assertions. They crash under test and in development, but log in production. Monitor the occurrences in production and alert if they go above some threshold rate. The assertions should have an owning team, and potentially DRI. This info should be a required argument to the soft-assert function. Otherwise you’re just making a mess. Hard assertions, for things that should never ever happen, or where crashing is preferable to continuing.
philipov 8y ago
The biggest problem with assertions in Python code is that they throw a generic AssertionError when you should be throwing a custom exception class instead. You can't catch the right error (and nothing but the right error) that way.
gnachman 8y ago
By default macOS apps do not terminate on certain assertions within certain event handlers, because exceptions are caught by the framework. The app just does weird stuff after this happens. This is terrible. Once I figured out how to make it crash I fixed a handful of bugs and the number of reports of inexplicable behavior went way down.
- saagarjha 8y ago
  I'm sure you're aware of this, but you can change this with NSSetUncaughtExceptionHandler: https://developer.apple.com/documentation/foundation/1409609-nssetuncaughtexceptionhandler https://developer.apple.com/documentation/foundation/1409609.... By default Cocoa registers an exception handler that swallows exceptions and asks the user whether they want to continue after the exception is raised.
neves 8y ago
It is important to note that the use of assertions is one of the few development practices that has empirical evidence support: https://www.microsoft.com/en-us/research/publication/assessing-the-relationship-between-software-assertions-and-code-qualityan-empirical-investigation/ https://www.microsoft.com/en-us/research/publication/assessi...
rkeene2 8y ago
In my view asserts are just a way of defining the contract of an interface that cannot be specified to the compiler/runtime any other way. They are a workaround of limitations of those systems, or rather the extend a system with infinite flexibility. For example the NaCl "randombytes" function has the prototype: void randombytes(unsigned char *buffer, unsigned long long length); It has no way to report an error -- its contract is that it must fill the buffer with high quality random bytes before returning. And the OpenBSD function "arc4random_buf" has the prototype: void arc4random_buf(void *buf, size_t nbytes); So we have a potential type difference between the two interfaces for the amount of data that we can generate. If you wanted to back randombytes() by arc4random_buf() you have several options: 1. Handle the case within randombytes() where the (unsigned long long) value exceeds the maximum value of (size_t) by making repeated calls to arc4random_buf(); 2. Assert, at compile time, that (size_t)'s range falls within (unsigned long long) and there can be no problem (since arc4random_buf() similarly cannot fail); 3. Redefine the contract of randombytes() such that specifying a buffer length that exceeds (size_t)'s maximum value is invalid -- this is really the same as #2 but instead of failing to compile if (SIZE_MAX < ULLONG_MAX) the program will compile and run fine as long as the contract is not violated; The assert in this case is a guard against the violation that should never occur and helps to enforce the contract during development With regards to production use of assert, catching a contract violation there means a lot of things have gone wrong. How likely things are to go wrong and how much damage they do when they do go wrong can vary from contract to contract. Various asserts may be used based on the such things, from compile-time to run-time debug to run-time production to an inline debugging framework. There are trade-offs and no answer for all cases.
jacquesm 8y ago
The larger a codebase the bigger the value of a crash 'close' to the root cause of the error. Assertions help you to get - sometimes substantially - closer to the root cause of a problem and any assertion that does not hold would lead to a crash - or worse, silent data corruption - any way, so by all means, leave them on.
United857 8y ago
The one time that it's actually a good idea to intentionally crash an app in production is if failing the assert could e.g. cause irreversible corruption of important data or worse, lead to a security vulnerability/exploit. e.g. situations where you're writing to a buffer/disk, dealing with raw pointers, etc. (which should hopefully shouldn't happen with good design, but sometimes unavoidable)
- WalterBright 8y ago
  My work is cut out for me :-(
WalterBright 8y ago
Author here. AMA. My other two articles on the topic: Safe Systems from Unreliable Parts https://www.digitalmars.com/articles/b39.html https://www.digitalmars.com/articles/b39.html Designing Safe Software Systems Part 2 https://www.digitalmars.com/articles/b40.html https://www.digitalmars.com/articles/b40.html
WalterBright 8y ago
If you want some fun (fun as in ruining your day, don't try this without backing up first) fill up your system disk to near capacity, then try to run various apps and system utilities. All kinds of bizarre things start happening, usually bad. The cause is simple, most of these are written in C and C programmers rarely check to see if writes to the disk succeed. For example, a few months ago, my Windows box would crash every time it would auto-update. I did a lot of cursing about this, as how could Microsoft do this? I eventually realized that the disk was nearly full. Cleared out a few gigs, and the auto-update started working. This is 2018. It's not specific to Windows, either. It happens with every OS I've ever tried it with, including Linux. No message like "disk full", or "failed to write file". Just erratic random behavior and weird things happening. I always try to keep at least 10% of disk free.
- tonyedgecombe 8y ago
  I suspect it's the same with memory, nobody checks that malloc doesn't return 0.
  - WalterBright 8y ago
    That's right, but at least these days you'll get seg fault if you try to use it. Back in the bad old DOS days, NULL pointers pointed at the interrupt dispatch table, so running out of memory (awfully common on a 640K machine) meant you trampled all over the operating system. It was so bad that I'd defensively reboot my machine constantly while debugging. DOS extenders saved the day, because they ran code in protected mode. I never developed code in real mode again, I just ported fully debugged code to it.
- jvanderbot 8y ago
  Your post is just as true with "The cause is simple, most programmers rarely check to see if writes to disk succeed", but is somewhat more credible.
  - TeMPOraL 8y ago
    In some other languages, a failed write will kill your application with an IO exception. Or at the very least (in Java) the exception will hit a handler that you are forced to write. The point being, those languages are designed to force a different code path on you in situations like failed IO, and people writing standard libraries make sure to check return values of C calls and throw exceptions if they're bad.
rdtsc 8y ago
> High reliability is not achieved by making perfect designs, it is achieved by making designs that are tolerant of failure. Runtime checking is essential to this, as when a fault is detected the program can go into a controlled state doing things like Very good point. That's is exactly the philosophy behind Erlang (the language and its VM, also extends to Elixir and other BEAM languages). It has isolated process heaps and process hierarchy supervisors. That is the most critical part of the deal because it means crashes and failures are controlled and only a small part of the system would be affected, without it spidering out and putting the rest of the system and put it in an unknown state. Microservices or just using OS processes instead of threads can kind of emulate that. But you can only have so many OS processes as they can pretty heavyweight. And with Microservices there whole other stack of stuff involved as opposed to just the language environment. > I find these options far preferable than going into an unknown state and praying. The idea of avoiding "unknown" state is key. That's also where crashing and restarting comes in - to get back to a known state. Of course you'd also want to log the failure so someone can eventually fix it. But, if a system is well designed, someone won't have to do it at 4am in the morning. I've seen systems crashing and restarting for weeks. Sounds terrible at first, but the alternative would have been having a service that's down completely and waking someone up in the middle of the night to fix the issue.
carapace 8y ago
Most people don't know that Python has a '-O' optimize CLI option: $ python -h usage: python [option] ... [-c cmd | -m mod | file | -] [arg] ... Options and arguments (and corresponding environment variables): ... -O : optimize generated bytecode slightly; also PYTHONOPTIMIZE=x -OO : remove doc-strings in addition to the -O optimizations ... It does two things (that I know of): 1.) Assertions are removed. The assertion code does not appear in the bytecode. 2.) Any code guarded by a "if __debug__:" statement is also removed. YMMV
WalterBright 8y ago
Famous Last Words: "It's just a buffer overflow, nothing to worry about."

In my code, most asserts become log messages in release builds. This helps a lot when troubleshooting problems, customers don't have debuggers installed. I often work on desktop software, unfortunately it's impossible to achieve avionics level of reliability because I don't have a luxury of controlled hardware and software environment.

gilgoomesh 8y ago

The article argues in favor of assertions in production code but acknowledges that the author's experience is from a very specific field with very specialized constraints and extensive design around recovering from aborted components. This advice is not obviously applicable to different kinds of programming with different constraints and different environments. As with so many things: one solution does not fit all problems.

Nomentatus 8y ago
You're gainsaying, but others have dropped empirical evidence on the other side, averring a low to no downside; and my experience is all on the other side too. It would help if you gave some lived examples.

sbradford26 8y ago

I think part of what is argued is that people need to look at their software and determine what failure modes are not acceptable. It is impossible to design a system that won't fail so the focus should be on designing parts that detect and handle failures. For those parts you may want to pull in techniques from aerospace such as command monitor setups, assertions, fail-safe modes, and force restarts. Clearly not all software is flight critical but a lot of software could be improved by treating some parts of it as such.

pmichalina 8y ago

Assertions, throws, and try-catch are code smells. Really bad design choice considering we have more elegant solutions like monads.

TeMPOraL 8y ago
Assertions, yes, in so far as they mean "I know this can't happen, but I can't structurally prevent it in the program in an elegant fashion, with the tools I use". However, I can't see how monads are suddenly going to make this problem disappear. As for throws/try-catch - beyond assertion use, they also report all problems that are outside your control (like IO failing due to HDD damage, or connection dropping). Those kinds of problems are here to stay.
oftenwrong 8y ago
Sure, monads provide an elegant way to structure code with error conditions, but I don't see a problem with using simple assertions, throws, try-catch to enforce invariants. Personally, I don't care about elegance as much as whether or not it works, as long as it is simple to use and maintain. I can return something like a Maybe/Either, or I can throw an exception - both approaches are fine by me if it forces the programmer to account for error cases.
- pmichalina 8y ago
  Throwing is a side effect breaking flow of control for no good reason. It’s takes you down the same path as all other spaghetti code. Not as much elegance as it is about purity. Minimizing side effects is the number one way to reduce bugs. It should be the number one guiding principle when creating out reliable software. Which means, you simply cannot use the primitive try catch or similar construct. Don’t break flow of control. Guide it to a terminal value instead. These days, when I see try-catch and if-else constructs (which is in most codebases) it’s clear there will be bugs over the life of the application. It’s fine, use them, but there is a world of greatness when you ditch these faulty constructs. Just like ditching OOP constructs. All built on false premises.
  - 8y ago

dougk16 8y ago

There's a good approach to asserts I adopted years ago for your typical low-risk CRUD/web/mobile app that I'm guessing ~90% of us are working on (as opposed to aviation software like in the article): Use asserts as you normally would, but in production have it fire an event to google analytics or some such. This is particularly useful for those middle-ground situations where a full app crash isn't justified because things will probably chug along fine, but nonetheless you want to know something's screwy. For debug/testing mode I usually pop a toast or dialog so the tester knows to report something to me even if the app is otherwise working fine. You often say to yourself "Well I know this assert will never trigger but I'll put it here just in case". Then you check your analytics next day after release and get humbled.

vram22 8y ago
>You often say to yourself "Well I know this assert will never trigger but I'll put it here just in case". Then you check your analytics next day after release and get humbled. That is exactly what asserts are for: checking for things that "cannot happen" - but do sometimes happen.

roland35 8y ago

Software development often does seem like a struggle between reliability/robustness and safety/correctness. Figuring out what to do with asserts falls in this debate! For embedded development I have often had ASSERTs log out to a serial port with the function name/line # and continue operating - this can then be connected to any logging device. This makes the system a little more robust (it won't crash, right away at least) but less correct/safe. During development testing we often had complaints of machines freezing due to asserts, which always increased the priority of those bugs! Definitely a good thing in the long run to fix those bugs and make the system more correct. Stopping all executing may not be the correct 'safe state' for an airplane though!

sbradford26 8y ago
Usually the hardware is tri or even quad redundant so the system can handle one of the boxes halting execution while it is either reset or other action is taken. If several failures happen one box may be powered off by the others to stop erroneous outputs from the failed box.
- WalterBright 8y ago
  May? The offending box gets electrically isolated from the craft if any fault is detected in it. Any fault. Now, the pilot may attempt to restart it by manipulating the circuit breakers, but that's the pilot's option.
WalterBright 8y ago
> Stopping all executing may not be the correct 'safe state' for an airplane though! It absolutely is. It's a (literally) disaster to allow code that has entered an unknown state to control aircraft functions. There is NO WAY such a design would EVER be certified by the FAA.
- cryptonector 8y ago
  In a fly-by-wire design (which is de rigeur now), stopping all operations would be way worse than continuing.

alangpierce 8y ago

At my work (hosted enterprise webapp), both the frontend and backend have both an `assert` function and an `assertAndContinue` function. `assert` always crashes on failure, and `assertAndContinue` crashes in dev and logs an error and keeps going in prod. Each time we want to verify a runtime assumption, we decide which type of assert to use. We prefer `assertAndContinue` (and I push for it in code review), but there are reasons to use plain assert: • In some cases, crashing is a better user experience than proceeding in an unknown state. Usually this is on the backend when data integrity is at risk, but could be on the frontend when we're at risk of making a server call with the wrong arguments. But usually, pretty much anything that doesn't crash is better than crashing. • `assertAndContinue` requires that you have a reasonable fallback. If code is going to crash the next line anyway, there's no point in `assertAndContinue`. In most cases, it's easy, but sometimes it takes real engineering, like building in an error state into the UI component that skips downstream code. When there isn't an obvious fallback, using `assertAndContinue` is a judgement call based on the difficulty of implementing a real fallback, the likelihood of the bug actually happening, and severity if the bug does trigger.

wvenable 8y ago
I have a difficult time imagining anytime that assertAndContinue makes sense. If a fallback is necessary and recovery possible, I would have asserts throw and use normal exceptional handling for that.
- MaulingMonkey 8y ago
  Exceptions are a control flow mechanism, not necessarily an error collection mechanism. I also frequently work with languages either without exception handling or with exception handling disabled for certain targets. Even when exception handling is enabled, it's a bit silly and heavyweight if you just need to short circuit a single conditional, loop, or function. Why throw when you can break or return? And when you do use nonfatal-but-logged exceptions, what do you put in the exception handler and what makes it different from assertAndContinue besides minor errata like being specialized for exceptions?

tedd4u 8y ago

I recently discussed this with an engineering manager for a notable consumer iOS app with tens of millions of downloads. In discussing how to better reduce their crash rate, there was a debate on his team about whether to enable assertions in their production iOS app. As any iOS developer knows, even if you have a great crash reporter and database there are a lot of crashes that are hard to figure out. One faction believed you should never intentionally crash the app, while another faction believed the app was going to crash anyways or maybe do something worse. They decided to test it out and during a sprint that was already mostly about bug fixes where they would do extra testing before release, they enabled asserts, and closely monitored their crash reporter. What they found was that the number of crashes did not change much, but the cardinality went down, significantly. The learning was code executing past a disabled assertion may be in one of n different bad states, each of which might lead to a different type of crash. They now had better high-level information about what was causing crashes (knowing which asserts were wrong) and it helped them reduce their crash rate much more than raw crashes without asserts (including cases where the crash was in iOS code, not app code).

bsaul 8y ago
That's a really interesting results. Those are the kind of thought you may have but knowing that it actually worked on a production app that's massively used is a really valuable thing. Anyone else had similar experience ?
- noir_lord 8y ago
  I inherited a hilarious bad(ly written) legacy system. One of the first things I did was turn on every possible "throw on error, die on error, assert" in development (could't do it production as it'd just fall over constantly). The constant crashes in dev meant I rapidly fixed the low hanging fruit (undefined indexes etc) and then slowly as I cleaned an area up I'd leave the asserts in once in production since at that point what I'm asserting is "I think I fixed all the insanity but tell me straight away if my assumption was bad". Painful very painful but those areas now cause me very few if any issues.
- pfranz 8y ago

js8 8y ago

I completely agree with the approach outlined in the original post. (It's also called defensive programming.) Not failing fast and hard when the application is in unexpected state (which is what assert should check) is robust only on the surface. I think we have the debate because it's not just about adding asserts. If you want to add asserts to existing codebase, essentially making it to fail fast and hard, you need to have a system architecture that is capable of dealing with said failure. If your codebase isn't architected as such, you will probably make things worse for the user by adding extra asserts. I think that's what the people who are against asserts fear of, and rightfully so. But the answer shouldn't be do not use asserts, the answer should be make the system more resilient to failure first.

dwohnitmok 8y ago

It doesn't fix everything, but judicious use of private data constructors and custom types can go a long way towards reducing the need for asserts. They can bubble up potential violations of preconditions and force handler code to be written. Some Scala examples, but they're hopefully quite language agnostic: https://m.youtube.com/watch?v=Csj3lzsr0_I https://m.youtube.com/watch?v=Csj3lzsr0_I https://m.youtube.com/watch?v=keTId618iOs https://m.youtube.com/watch?v=keTId618iOs

draw_down 8y ago

Soft assertions. They crash under test and in development, but log in production. Monitor the occurrences in production and alert if they go above some threshold rate. The assertions should have an owning team, and potentially DRI. This info should be a required argument to the soft-assert function. Otherwise you’re just making a mess. Hard assertions, for things that should never ever happen, or where crashing is preferable to continuing.

philipov 8y ago

The biggest problem with assertions in Python code is that they throw a generic AssertionError when you should be throwing a custom exception class instead. You can't catch the right error (and nothing but the right error) that way.

gnachman 8y ago

By default macOS apps do not terminate on certain assertions within certain event handlers, because exceptions are caught by the framework. The app just does weird stuff after this happens. This is terrible. Once I figured out how to make it crash I fixed a handful of bugs and the number of reports of inexplicable behavior went way down.

saagarjha 8y ago
I'm sure you're aware of this, but you can change this with NSSetUncaughtExceptionHandler: https://developer.apple.com/documentation/foundation/1409609-nssetuncaughtexceptionhandler https://developer.apple.com/documentation/foundation/1409609.... By default Cocoa registers an exception handler that swallows exceptions and asks the user whether they want to continue after the exception is raised.

neves 8y ago

It is important to note that the use of assertions is one of the few development practices that has empirical evidence support: https://www.microsoft.com/en-us/research/publication/assessing-the-relationship-between-software-assertions-and-code-qualityan-empirical-investigation/ https://www.microsoft.com/en-us/research/publication/assessi...

rkeene2 8y ago

In my view asserts are just a way of defining the contract of an interface that cannot be specified to the compiler/runtime any other way. They are a workaround of limitations of those systems, or rather the extend a system with infinite flexibility. For example the NaCl "randombytes" function has the prototype: void randombytes(unsigned char *buffer, unsigned long long length); It has no way to report an error -- its contract is that it must fill the buffer with high quality random bytes before returning. And the OpenBSD function "arc4random_buf" has the prototype: void arc4random_buf(void *buf, size_t nbytes); So we have a potential type difference between the two interfaces for the amount of data that we can generate. If you wanted to back randombytes() by arc4random_buf() you have several options: 1. Handle the case within randombytes() where the (unsigned long long) value exceeds the maximum value of (size_t) by making repeated calls to arc4random_buf(); 2. Assert, at compile time, that (size_t)'s range falls within (unsigned long long) and there can be no problem (since arc4random_buf() similarly cannot fail); 3. Redefine the contract of randombytes() such that specifying a buffer length that exceeds (size_t)'s maximum value is invalid -- this is really the same as #2 but instead of failing to compile if (SIZE_MAX < ULLONG_MAX) the program will compile and run fine as long as the contract is not violated; The assert in this case is a guard against the violation that should never occur and helps to enforce the contract during development With regards to production use of assert, catching a contract violation there means a lot of things have gone wrong. How likely things are to go wrong and how much damage they do when they do go wrong can vary from contract to contract. Various asserts may be used based on the such things, from compile-time to run-time debug to run-time production to an inline debugging framework. There are trade-offs and no answer for all cases.

jacquesm 8y ago

The larger a codebase the bigger the value of a crash 'close' to the root cause of the error. Assertions help you to get - sometimes substantially - closer to the root cause of a problem and any assertion that does not hold would lead to a crash - or worse, silent data corruption - any way, so by all means, leave them on.

United857 8y ago

The one time that it's actually a good idea to intentionally crash an app in production is if failing the assert could e.g. cause irreversible corruption of important data or worse, lead to a security vulnerability/exploit. e.g. situations where you're writing to a buffer/disk, dealing with raw pointers, etc. (which should hopefully shouldn't happen with good design, but sometimes unavoidable)

WalterBright 8y ago
My work is cut out for me :-(

WalterBright 8y ago

Author here. AMA. My other two articles on the topic: Safe Systems from Unreliable Parts https://www.digitalmars.com/articles/b39.html https://www.digitalmars.com/articles/b39.html Designing Safe Software Systems Part 2 https://www.digitalmars.com/articles/b40.html https://www.digitalmars.com/articles/b40.html

WalterBright 8y ago

If you want some fun (fun as in ruining your day, don't try this without backing up first) fill up your system disk to near capacity, then try to run various apps and system utilities. All kinds of bizarre things start happening, usually bad. The cause is simple, most of these are written in C and C programmers rarely check to see if writes to the disk succeed. For example, a few months ago, my Windows box would crash every time it would auto-update. I did a lot of cursing about this, as how could Microsoft do this? I eventually realized that the disk was nearly full. Cleared out a few gigs, and the auto-update started working. This is 2018. It's not specific to Windows, either. It happens with every OS I've ever tried it with, including Linux. No message like "disk full", or "failed to write file". Just erratic random behavior and weird things happening. I always try to keep at least 10% of disk free.

tonyedgecombe 8y ago
I suspect it's the same with memory, nobody checks that malloc doesn't return 0.
- WalterBright 8y ago
  That's right, but at least these days you'll get seg fault if you try to use it. Back in the bad old DOS days, NULL pointers pointed at the interrupt dispatch table, so running out of memory (awfully common on a 640K machine) meant you trampled all over the operating system. It was so bad that I'd defensively reboot my machine constantly while debugging. DOS extenders saved the day, because they ran code in protected mode. I never developed code in real mode again, I just ported fully debugged code to it.
jvanderbot 8y ago
Your post is just as true with "The cause is simple, most programmers rarely check to see if writes to disk succeed", but is somewhat more credible.
- TeMPOraL 8y ago
  In some other languages, a failed write will kill your application with an IO exception. Or at the very least (in Java) the exception will hit a handler that you are forced to write. The point being, those languages are designed to force a different code path on you in situations like failed IO, and people writing standard libraries make sure to check return values of C calls and throw exceptions if they're bad.

rdtsc 8y ago

> High reliability is not achieved by making perfect designs, it is achieved by making designs that are tolerant of failure. Runtime checking is essential to this, as when a fault is detected the program can go into a controlled state doing things like Very good point. That's is exactly the philosophy behind Erlang (the language and its VM, also extends to Elixir and other BEAM languages). It has isolated process heaps and process hierarchy supervisors. That is the most critical part of the deal because it means crashes and failures are controlled and only a small part of the system would be affected, without it spidering out and putting the rest of the system and put it in an unknown state. Microservices or just using OS processes instead of threads can kind of emulate that. But you can only have so many OS processes as they can pretty heavyweight. And with Microservices there whole other stack of stuff involved as opposed to just the language environment. > I find these options far preferable than going into an unknown state and praying. The idea of avoiding "unknown" state is key. That's also where crashing and restarting comes in - to get back to a known state. Of course you'd also want to log the failure so someone can eventually fix it. But, if a system is well designed, someone won't have to do it at 4am in the morning. I've seen systems crashing and restarting for weeks. Sounds terrible at first, but the alternative would have been having a service that's down completely and waking someone up in the middle of the night to fix the issue.

carapace 8y ago

Most people don't know that Python has a '-O' optimize CLI option: $ python -h usage: python [option] ... [-c cmd | -m mod | file | -] [arg] ... Options and arguments (and corresponding environment variables): ... -O : optimize generated bytecode slightly; also PYTHONOPTIMIZE=x -OO : remove doc-strings in addition to the -O optimizations ... It does two things (that I know of): 1.) Assertions are removed. The assertion code does not appear in the bytecode. 2.) Any code guarded by a "if __debug__:" statement is also removed. YMMV

WalterBright 8y ago

Famous Last Words: "It's just a buffer overflow, nothing to worry about."