4 ms·
A few things: you usually rollback quickly. After minutes, not weeks. Defense in depth. Backups, architecture that's supports replay etc. Dogfooding. No one w
by joshuamorton 6y ago
A few things: you usually rollback quickly. After minutes, not weeks.
Defense in depth. Backups, architecture that's supports replay etc.
Dogfooding. No one will complain if you're the person who wrote the code that lost your emails.
Make changes backwards compatible when possible. Add a new column. Wait a while. Write data to the column. You can rollback cleanly without issue at each step. Naturally backwards compatible schemas (protos) help with this.
Dual writing. Similar to the above, if you're replacing a with b, dual write a and b for a while. Validate everything. Start relying only on b, but keep writing a. Eventually stop writing a after a while. At every point, you can roll back to the previous state without issue, so if you can validate everything functions at each interim state, you're always able to roll back.
I'm sure there's more I'm forgetting
- kqr 6y agoThis is actually all very good advice and succinctly put at that. If you think of more, please do add it to your comment!
- Darkstryder 6y agoThank you for the very relevant advice. All of these suggestions are certainly feasible but taken together they do amount to a non-negligible cost in infrastructure and development velocity though. Where I live I've yet to find an employer that can be convinced these costs are worth it.
- kqr 6y agoThis is one of those "slow is smooth; smooth is fast" kinds of situations. Frequent shipments is correlated with high developer performance (for reasons I haven't time to get into, but it boils down to feedback and motivation). Frequent shipments is only possible when taking tiny steps at a time. In the end, you save time and money by doing it that way. If your employer does not understand that, it is part of your professional responsibility to try to convince them of it.