22 ms·

Rolling back to a known-good state is the safe approach Absolutely. However, they rollback should be atomic, which means all pieces of the infrastructure/code

Rolling back to a known-good state is the safe approach Absolutely. However, they rollback should be atomic, which means all pieces of the infrastructure/code should be rolled back to a known-good-state. When I said "gymnastics", I meant for rolling back one piece of code, only to find incompatibility with other pieces. I do not intend to judge them, knowing its difficult not to panic in the difficult emergency situation, but not working effect out on paper (or not knowing, which versions of software components are inter-compatible), before actually doing it on code, look pretty novice to me, for a company of heroku's scale. I really hope this is not an unfair criticism. (handling emergencies are difficult)

erikpukinskis 16y ago
The trouble is that they have messaging servers, a routing mesh, and caching servers which are all loosely coupled and deployed on separate boxes. They could take down all dozens of pieces of their infrastructure and roll them all back to wherever they were on that previous date, but this is not better for several reasons: 1) it take much longer than just rebooting the isolated service. Can you imagine Google shutting down every one of their multi-million boxes, rolling them back to a previous state and spinning them up again? 2) they'd still be at risk for incompatibilities with their databases, etc. the problem with unexpected imcomptabilities is that they're unexpected.
- random42 16y ago
  All I am trying to say, it only makes sense to do/know effect analysis of your changes, before actually doing it. I fail to see, why they need to revert a piece of code, and then realize, OMG... this version of code does not fit well with the rest of architecture, now change it back. 1.) I do not expect Google (or even a small shop, like my place) to revert any piece of code which is not affected. I, HOWEVER, expect to know what changes I am EXACTLY doing, and what to EXPECT after the changes. (It should not be black magic, for historical code). 2.) I fail to understand this, why should this the case for older code? I can understand some tricky/edge/minor cases, but whether the architecture/database etc. (major compatibility) is compatible or not, should be possible to calculate BEFORE doing the changes. I hope I am not over-trivializing the issue, but I still cannot get my head over the approach.

erikpukinskis 16y ago

The trouble is that they have messaging servers, a routing mesh, and caching servers which are all loosely coupled and deployed on separate boxes. They could take down all dozens of pieces of their infrastructure and roll them all back to wherever they were on that previous date, but this is not better for several reasons: 1) it take much longer than just rebooting the isolated service. Can you imagine Google shutting down every one of their multi-million boxes, rolling them back to a previous state and spinning them up again? 2) they'd still be at risk for incompatibilities with their databases, etc. the problem with unexpected imcomptabilities is that they're unexpected.

random42 16y ago
All I am trying to say, it only makes sense to do/know effect analysis of your changes, before actually doing it. I fail to see, why they need to revert a piece of code, and then realize, OMG... this version of code does not fit well with the rest of architecture, now change it back. 1.) I do not expect Google (or even a small shop, like my place) to revert any piece of code which is not affected. I, HOWEVER, expect to know what changes I am EXACTLY doing, and what to EXPECT after the changes. (It should not be black magic, for historical code). 2.) I fail to understand this, why should this the case for older code? I can understand some tricky/edge/minor cases, but whether the architecture/database etc. (major compatibility) is compatible or not, should be possible to calculate BEFORE doing the changes. I hope I am not over-trivializing the issue, but I still cannot get my head over the approach.