4 ms·
> the recommendation at places like Google is to rollback first and investigate the problem later I always have the same question regarding this advice: how ar
by Darkstryder 6y ago
> the recommendation at places like Google is to rollback first and investigate the problem later
I always have the same question regarding this advice: how are you supposed to handle data corruption?
When a buggy new version creates lasting problems in the persistent data, reverting to the last known good version may not fix the problem as the correct code now has to deal with incorrect data, which lead to a different, possibly incorrect behavior compared to before the buggy deployment.
Reverting the data itself is often not possible as you usually can’t say to your customers « woops, we deleted all of your bank transactions / incoming emails for the last two weeks because of a rollback on our side ».
So in my experience you still have to roll forward in a lot of cases, to fix both the code and data together.
How do you guys handle that in practice?
- saberdancer 6y agoWhat is different between rollback and manual data fixing or roll forward and manual data fixing? Well, rollback is usually faster and safer. In some cases you do not have data corruption and can roll back safely. If not, you can probably be quicker to fix the data than find the solution for the underlining problem.
- zorked 6y agoRollbacks are easy and simple to do and "solve" a large number of problems and go a long way bisecting the problem. They are not expected to solve every problem.
- joshuamorton 6y agoA few things: you usually rollback quickly. After minutes, not weeks. Defense in depth. Backups, architecture that's supports replay etc. Dogfooding. No one will complain if you're the person who wrote the code that lost your emails. Make changes backwards compatible when possible. Add a new column. Wait a while. Write data to the column. You can rollback cleanly without issue at each step. Naturally backwards compatible schemas (protos) help with this. Dual writing. Similar to the above, if you're replacing a with b, dual write a and b for a while. Validate everything. Start relying only on b, but keep writing a. Eventually stop writing a after a while. At every point, you can roll back to the previous state without issue, so if you can validate everything functions at each interim state, you're always able to roll back. I'm sure there's more I'm forgetting
- kqr 6y agoThis is actually all very good advice and succinctly put at that. If you think of more, please do add it to your comment!
- Darkstryder 6y agoThank you for the very relevant advice. All of these suggestions are certainly feasible but taken together they do amount to a non-negligible cost in infrastructure and development velocity though. Where I live I've yet to find an employer that can be convinced these costs are worth it.
- kqr 6y agoThis is one of those "slow is smooth; smooth is fast" kinds of situations. Frequent shipments is correlated with high developer performance (for reasons I haven't time to get into, but it boils down to feedback and motivation). Frequent shipments is only possible when taking tiny steps at a time. In the end, you save time and money by doing it that way. If your employer does not understand that, it is part of your professional responsibility to try to convince them of it.