2 ms·

This is not a professional mentality. Releasing later is better than introducing the bug again, never mind a new one.

by wbkang 9y ago

alangpierce 9y ago
A professional mentality is to do a risk analysis based on what you know and based on reasonable estimates of what you don't know. Here's how I'd model it. The main decision point on Apple's part is how long to QA the fix before pushing the release out. As soon as the build finishes? 3 hours of testing? 1 day? 3 days? 2 weeks? The longer you test, the lower the probability of introducing a regression or having an incomplete fix, but that probability never drops to zero. You could come up with an "expected after-patch defect severity over time" graph that goes down (due to more QA) in a way similar to exponential decay, and compare with that with the known before-patch severity of the root access bug (which was extremely bad in this case, obviously). A simple way to make the "how long to test" decision is to say that if the expected after-patch severity is lower than the known before-patch severity, then you should release. (That doesn't mean you need to stop testing, obviously.) A more nuanced decision-making process would take into account the confidence level from the QA process, the overhead of the release process, and the bad PR of releasing a buggy fix. But still, if the software has a terrible flaw and you have a fix that you're reasonably confident won't make things worse, it probably makes sense to release it. Apple ended up taking about a day to release a fix. My understanding is they also released it as a user-initiated update for a while, then later forced the update on everyone, which seems like a fine strategy to reduce risk of widespread regressions. Inventing a rating system, I'd say the severity of the root bug was 100 and the severity of the file sharing bug was 10, so the patch they released was an improvement even to users who ran into the file sharing bug. So over the course of multiple patches, their computers got better, even if was from terrible to bad to normal. I also think it's not obvious that another day of testing would have caught the file sharing bug, and taking 2 days to release a critical security fix is much worse than taking 1 day, in my opinion. So my impression is Apple generally made the right choices about how to release the patch, even if it's crappy that there was a regression and that such a bad security bug would exist in the first place.

This is not a professional mentality. Releasing later is better than introducing the bug again, never mind a new one.

by wbkang 9y ago

This is not a professional mentality. Releasing later is better than introducing the bug again, never mind a new one.

A professional mentality is to do a risk analysis based on what you know and based on reasonable estimates of what you don't know. Here's how I'd model it. The main decision point on Apple's part is how long to QA the fix before pushing the release out. As soon as the build finishes? 3 hours of testing? 1 day? 3 days? 2 weeks? The longer you test, the lower the probability of introducing a regression or having an incomplete fix, but that probability never drops to zero. You could come up with an "expected after-patch defect severity over time" graph that goes down (due to more QA) in a way similar to exponential decay, and compare with that with the known before-patch severity of the root access bug (which was extremely bad in this case, obviously). A simple way to make the "how long to test" decision is to say that if the expected after-patch severity is lower than the known before-patch severity, then you should release. (That doesn't mean you need to stop testing, obviously.) A more nuanced decision-making process would take into account the confidence level from the QA process, the overhead of the release process, and the bad PR of releasing a buggy fix. But still, if the software has a terrible flaw and you have a fix that you're reasonably confident won't make things worse, it probably makes sense to release it. Apple ended up taking about a day to release a fix. My understanding is they also released it as a user-initiated update for a while, then later forced the update on everyone, which seems like a fine strategy to reduce risk of widespread regressions. Inventing a rating system, I'd say the severity of the root bug was 100 and the severity of the file sharing bug was 10, so the patch they released was an improvement even to users who ran into the file sharing bug. So over the course of multiple patches, their computers got better, even if was from terrible to bad to normal. I also think it's not obvious that another day of testing would have caught the file sharing bug, and taking 2 days to release a critical security fix is much worse than taking 1 day, in my opinion. So my impression is Apple generally made the right choices about how to release the patch, even if it's crappy that there was a regression and that such a bad security bug would exist in the first place.