3 ms·

I recently discussed this with an engineering manager for a notable consumer iOS app with tens of millions of downloads. In discussing how to better reduce thei

by tedd4u 8y ago

I recently discussed this with an engineering manager for a notable consumer iOS app with tens of millions of downloads. In discussing how to better reduce their crash rate, there was a debate on his team about whether to enable assertions in their production iOS app. As any iOS developer knows, even if you have a great crash reporter and database there are a lot of crashes that are hard to figure out. One faction believed you should never intentionally crash the app, while another faction believed the app was going to crash anyways or maybe do something worse. They decided to test it out and during a sprint that was already mostly about bug fixes where they would do extra testing before release, they enabled asserts, and closely monitored their crash reporter. What they found was that the number of crashes did not change much, but the cardinality went down, significantly. The learning was code executing past a disabled assertion may be in one of n different bad states, each of which might lead to a different type of crash. They now had better high-level information about what was causing crashes (knowing which asserts were wrong) and it helped them reduce their crash rate much more than raw crashes without asserts (including cases where the crash was in iOS code, not app code).

bsaul 8y ago
That's a really interesting results. Those are the kind of thought you may have but knowing that it actually worked on a production app that's massively used is a really valuable thing. Anyone else had similar experience ?
- noir_lord 8y ago
  I inherited a hilarious bad(ly written) legacy system. One of the first things I did was turn on every possible "throw on error, die on error, assert" in development (could't do it production as it'd just fall over constantly). The constant crashes in dev meant I rapidly fixed the low hanging fruit (undefined indexes etc) and then slowly as I cleaned an area up I'd leave the asserts in once in production since at that point what I'm asserting is "I think I fixed all the insanity but tell me straight away if my assumption was bad". Painful very painful but those areas now cause me very few if any issues.
- pfranz 8y ago
  I work in visual effects for film writing internal tools. Usually they're tools that take data from department to department or the department uses internally; fairly short processes that serialize data in between. One place I worked had a team that was very adamant about not really having much error checking. Not much of any qc process, either. Wait for someone to complain about bad data and respond. Honestly, this worked really well for small, skunkworks type projects that needed to be nimble. As you would expect, when errors did happen it was because of bad data from further up. You really had to know the system well to be productive (the cynical of us thought the developers liked this because they could look like heroes). I prefer to error early and clearly and I got a lot of push back. To be fair to them, often times the errors would be irrelevant to the specific thing they needed and it would have been preferable to ignore and carry on. It felt like I was imposing a bunch of bureaucracy and yak shaving. But bringing on new people and scaling the amount of data going through would have been impossible without more structure.
  - WalterBright 8y ago
    I find it sad that there seems to be no conventional wisdom in the software business about the proper way to do this. Everyone starts from scratch and tries to learn it the hard way.
taneq 8y ago
Well yeah, if an assert() isn't bogus, then anything that follows after it is going to be wrong in some way. In many contexts, crashing is the least worst thing you can do in this situation. If you're really working on something that HAS to keep running NO MATTER WHAT then you probably want to do the hardcore critical systems stuff like N independently developed systems voting on the outcome. (Ah, but developed to what spec? Now we've just moved the bugs from the code to the spec... At least hopefully there are fewer of them.)

by tedd4u 8y ago

bsaul 8y ago

That's a really interesting results. Those are the kind of thought you may have but knowing that it actually worked on a production app that's massively used is a really valuable thing. Anyone else had similar experience ?

noir_lord 8y ago
I inherited a hilarious bad(ly written) legacy system. One of the first things I did was turn on every possible "throw on error, die on error, assert" in development (could't do it production as it'd just fall over constantly). The constant crashes in dev meant I rapidly fixed the low hanging fruit (undefined indexes etc) and then slowly as I cleaned an area up I'd leave the asserts in once in production since at that point what I'm asserting is "I think I fixed all the insanity but tell me straight away if my assumption was bad". Painful very painful but those areas now cause me very few if any issues.
pfranz 8y ago
I work in visual effects for film writing internal tools. Usually they're tools that take data from department to department or the department uses internally; fairly short processes that serialize data in between. One place I worked had a team that was very adamant about not really having much error checking. Not much of any qc process, either. Wait for someone to complain about bad data and respond. Honestly, this worked really well for small, skunkworks type projects that needed to be nimble. As you would expect, when errors did happen it was because of bad data from further up. You really had to know the system well to be productive (the cynical of us thought the developers liked this because they could look like heroes). I prefer to error early and clearly and I got a lot of push back. To be fair to them, often times the errors would be irrelevant to the specific thing they needed and it would have been preferable to ignore and carry on. It felt like I was imposing a bunch of bureaucracy and yak shaving. But bringing on new people and scaling the amount of data going through would have been impossible without more structure.
- WalterBright 8y ago
  I find it sad that there seems to be no conventional wisdom in the software business about the proper way to do this. Everyone starts from scratch and tries to learn it the hard way.

taneq 8y ago

Well yeah, if an assert() isn't bogus, then anything that follows after it is going to be wrong in some way. In many contexts, crashing is the least worst thing you can do in this situation. If you're really working on something that HAS to keep running NO MATTER WHAT then you probably want to do the hardcore critical systems stuff like N independently developed systems voting on the outcome. (Ah, but developed to what spec? Now we've just moved the bugs from the code to the spec... At least hopefully there are fewer of them.)