4 ms·

Let's suppose that your software currently has an incident where the service is degraded. For every day you don't push a fix, you lose 0.X% of revenue. Some en

Let's suppose that your software currently has an incident where the service is degraded. For every day you don't push a fix, you lose 0.X% of revenue. Some engineer creates a fix by turning off some recently added experiment responsible for the service degradation. The rollback passes all unit tests and integration tests. You know this fix is low-risk (maybe because your service was healthy for the ~1 year prior to the rollout). Even if there is some risk that this rollback breaks the service at some point in the future, in comparison to the known revenue loss the expected loss of that breakage would be minimal. Another engineer objects to the rollback saying that "they don't know of the long term effects of such a rollback." You try to counter their claims by showing that previous rollbacks were almost always successful, and when they caused another failure that failure was not nearly as bad as the original service degradation. Also, the rollback passes all tests. That engineer then repeats "we don't know of the long term effects of this particular rollback, and I'm skeptical that the fix is good because the failing experiment and rollback was identified so quickly." (Rollback vs. fix-forward doesn't actually matter; the point is that all available knowledge shows that the fix is low risk, historically supported, and that there is likely no mechanism for random failure in 6+ months.) Do you think it would be correct to prefer the second engineer's claims?

ifyoubuildit 5y ago
Yeah, I get your point, at the end of the day the net result is what we care about. But the assumptions that you use to calculate that net are where all of the disagreements stem from. In your example, you've constrained it to losing well defined revenue, and "you know this fix is low risk", so the answer is easy. Maybe (wrongly or rightly) you don't have faith in the revenue numbers, or in the test suite or the engineers writing the tests or the fix, maybe there's office politics involved, etc. This thread started with my reaction to these lines from op: > Knowing long-term effects (1+ years) is not a precondition of approval for most vaccines. It just so happens that vaccines usually take a long time for approval, so we end up knowing long-term effects as a side effect. This is because there generally isn't a known mechanism for vaccines to have long term effects. Not ruling out the possibility, but we understand how vaccines work pretty well. [...] I've seen this claim dozens of times, and it always always always makes me think of those scenarios where you find a bug in code that you or someone you trust/respect wrote from the ground up, and you wonder "how in the f*** did this ever work?". I'm not claiming I'm right to feel that way, maybe this is a defect in my character, but I just can't buy the "we understand vaccines pretty well" thing.
- joshuamorton 5y ago
  I think the place where the disagreement or misunderstanding lies is in how we're describing thinking about unknowns. You seem to be taking this from the perspective that, because there are unknowns, literally anything could happen. And we don't think that's the case with my vaccines or code. Pushing v2 of your webapp to master isn't going to cause rockets to launch. There are things that could happen and there are even very surprising things that are unlikely but could happen. But not everything is possible. When we say "we understand how vaccines work pretty well", we mean that in the same was as "we know how our app works pretty well". That doesn't mean that we can't still be surprised, but it does mean that we have some things that we can be confident won't happen. One such thing is sudden onset symptoms well after the thing has left your system, because there isn't a biological way for that to have an effect (to use the code analogy, you enable an experiment, it modifies database writes for a while, then you disable the experiment and delete the code. Once you've deleted the code, we know it can't do anything else, because it's not there anymore. That's true with both vaccines and webapps.
  - ifyoubuildit 5y ago
    > One such thing is sudden onset symptoms well after the thing has left your system, because there isn't a biological way for that to have an effect The hard part about conversations like this are that you could have written the definitive book on this stuff, or you could be just another dummy on the internet like me. So I have to take a statement like that and do my best to evaluate it myself, since I have no idea how to value your expertise. If you're an expert of some sort, I apologize in advance. We know for a fact that vaccines have long term effects. They wouldn't be useful otherwise. It would be nice if they could only have positive long term effects, but why would there be any guarantee of that? Symptoms could start immediately but be undetectable until they progress far enough (think a tiny little tumor for just one example). Again, not claiming this is happening, just saying that it seems nutty to assume it can't happen. When I read "we understand how vaccines work pretty well", I automatically translate it to "we understand pretty well what happens when we inject a substance into a human body", and then I think "really?". We have some useful models, sure, but "understanding pretty well" seems a vast overstatement of the situation, especially when its laypeople chatting on the internet. The abstract concept of a vaccine is pretty simple. Anybody can regurgitate the picture book version of it. Any specific instantiation of one is remarkably complex, all the way from manufacturing, to transport, to administration, and then interacting with the legacy ball of spaghetti that is the human body and it's various systems. The fact that we humans decided to name some substance the same thing that we named some other substances that we created in the past doesn't necessarily tell you how that substance is going to interact with the body. It can provide you with hypotheses, but hypotheses need to be tested.

ifyoubuildit 5y ago

Yeah, I get your point, at the end of the day the net result is what we care about. But the assumptions that you use to calculate that net are where all of the disagreements stem from. In your example, you've constrained it to losing well defined revenue, and "you know this fix is low risk", so the answer is easy. Maybe (wrongly or rightly) you don't have faith in the revenue numbers, or in the test suite or the engineers writing the tests or the fix, maybe there's office politics involved, etc. This thread started with my reaction to these lines from op: > Knowing long-term effects (1+ years) is not a precondition of approval for most vaccines. It just so happens that vaccines usually take a long time for approval, so we end up knowing long-term effects as a side effect. This is because there generally isn't a known mechanism for vaccines to have long term effects. Not ruling out the possibility, but we understand how vaccines work pretty well. [...] I've seen this claim dozens of times, and it always always always makes me think of those scenarios where you find a bug in code that you or someone you trust/respect wrote from the ground up, and you wonder "how in the f*** did this ever work?". I'm not claiming I'm right to feel that way, maybe this is a defect in my character, but I just can't buy the "we understand vaccines pretty well" thing.

joshuamorton 5y ago
I think the place where the disagreement or misunderstanding lies is in how we're describing thinking about unknowns. You seem to be taking this from the perspective that, because there are unknowns, literally anything could happen. And we don't think that's the case with my vaccines or code. Pushing v2 of your webapp to master isn't going to cause rockets to launch. There are things that could happen and there are even very surprising things that are unlikely but could happen. But not everything is possible. When we say "we understand how vaccines work pretty well", we mean that in the same was as "we know how our app works pretty well". That doesn't mean that we can't still be surprised, but it does mean that we have some things that we can be confident won't happen. One such thing is sudden onset symptoms well after the thing has left your system, because there isn't a biological way for that to have an effect (to use the code analogy, you enable an experiment, it modifies database writes for a while, then you disable the experiment and delete the code. Once you've deleted the code, we know it can't do anything else, because it's not there anymore. That's true with both vaccines and webapps.
- ifyoubuildit 5y ago
  > One such thing is sudden onset symptoms well after the thing has left your system, because there isn't a biological way for that to have an effect The hard part about conversations like this are that you could have written the definitive book on this stuff, or you could be just another dummy on the internet like me. So I have to take a statement like that and do my best to evaluate it myself, since I have no idea how to value your expertise. If you're an expert of some sort, I apologize in advance. We know for a fact that vaccines have long term effects. They wouldn't be useful otherwise. It would be nice if they could only have positive long term effects, but why would there be any guarantee of that? Symptoms could start immediately but be undetectable until they progress far enough (think a tiny little tumor for just one example). Again, not claiming this is happening, just saying that it seems nutty to assume it can't happen. When I read "we understand how vaccines work pretty well", I automatically translate it to "we understand pretty well what happens when we inject a substance into a human body", and then I think "really?". We have some useful models, sure, but "understanding pretty well" seems a vast overstatement of the situation, especially when its laypeople chatting on the internet. The abstract concept of a vaccine is pretty simple. Anybody can regurgitate the picture book version of it. Any specific instantiation of one is remarkably complex, all the way from manufacturing, to transport, to administration, and then interacting with the legacy ball of spaghetti that is the human body and it's various systems. The fact that we humans decided to name some substance the same thing that we named some other substances that we created in the past doesn't necessarily tell you how that substance is going to interact with the body. It can provide you with hypotheses, but hypotheses need to be tested.