3 ms·

> They (apparently) didn't had code reviews I don't get that. There was no code issue. The old and new code both worked as intended, it was a deployment and de

> They (apparently) didn't had code reviews I don't get that. There was no code issue. The old and new code both worked as intended, it was a deployment and deployment-verification problem. > They didn't had a staging environment Yes they did. They staged the new code and tested it. They did a slow deployment also. > They didn't had a contingency plan to revert the deploy They did revert the deploy within the 45 minutes. It made it worse. I think you need to re-read the article. Your assessment is strange given the event.

paulrademacher 12y ago
Code review could have been another set of eyes to predict the problem of reusing a flag.
- solarmist 12y ago
  If the message was as compact and low level as possible it was probably a bit flag, so in that context it makes sense to repurpose it. Being so removed from binary and bit level interactions it can be easy to forget things like this.
- saalweachter 12y ago
  I agree with the GP; I don't think code reviews or testing was the problem. I think the best-practices they violated is that they deprecated and repurposed a flag within a single release cycle. That sort of activity should take two release cycles at least, one to remove the old functionality and one to add the new functionality.
hcarvalhoalves 12y ago
> I don't get that. There was no code issue. The old and new code both worked as intended, it was a deployment and deployment-verification problem. A code review could raise the issue of repurposing a flag in case they had to revert the deploy. Changing the semantics of a flag is a big no-no anyway, and there are ways to guard against that. > Yes they did. They staged the new code and tested it. They did a slow deployment also. But they didn't had a staging environment that matched their live environment, apparently. You want a staging environment that is 1:1. > They did revert the deploy within the 45 minutes. It made it worse. If you think reverting a deploy by simply pushing an older version is the same as a contingency plan, think again.

paulrademacher 12y ago

Code review could have been another set of eyes to predict the problem of reusing a flag.

solarmist 12y ago
If the message was as compact and low level as possible it was probably a bit flag, so in that context it makes sense to repurpose it. Being so removed from binary and bit level interactions it can be easy to forget things like this.
saalweachter 12y ago
I agree with the GP; I don't think code reviews or testing was the problem. I think the best-practices they violated is that they deprecated and repurposed a flag within a single release cycle. That sort of activity should take two release cycles at least, one to remove the old functionality and one to add the new functionality.

hcarvalhoalves 12y ago

> I don't get that. There was no code issue. The old and new code both worked as intended, it was a deployment and deployment-verification problem. A code review could raise the issue of repurposing a flag in case they had to revert the deploy. Changing the semantics of a flag is a big no-no anyway, and there are ways to guard against that. > Yes they did. They staged the new code and tested it. They did a slow deployment also. But they didn't had a staging environment that matched their live environment, apparently. You want a staging environment that is 1:1. > They did revert the deploy within the 45 minutes. It made it worse. If you think reverting a deploy by simply pushing an older version is the same as a contingency plan, think again.