5 ms·
Continuous deployment sounds right to me in principle, but I have yet to see a system to handle rollback in the face of database handling (one slide mentioned t
by cdavid 15y ago
Continuous deployment sounds right to me in principle, but I have yet to see a system to handle rollback in the face of database handling (one slide mentioned this briefly). When you have a bug (say you delete the wrong data), how do you rollback this ?
The only solution I can see is one where databases are handled differently (they generally are anyway at the deployment stage), and IMO, this is the challenging issue. Continuously deploying app servers is not trivial, but I would consider it a mostly solved problem.
- swanson 15y agoWhile it doesn't handle every case, using a database migration framework with Up/Down style methods as part of your deploy process can work pretty effectively. Another option is to use a VM for your database and do a snapshot before deploying.
- andrewvc 15y agoWell, yeah, but if you perform financial transactions, you can't just lose 5 minutes of history. Also, for a DB, LVM's probably just as good as a full VM.
- mathgladiator 15y agoChances are good that important tables do do transactions don't change as often as other things in the system. This is why I support things like CouchDB to handle the non-transaction stuff.
- nknight 15y agoBased on a very brief conversation I had with a Paypal engineer a few years back, it's SOP in modern financial transaction handling to store/log details of every transaction in many places and various forms, and there's plenty of (probably excessive) paranoia involved. The context of the discussion was performance, scalability, and engineering resources, so we didn't get into how the logs are used, but it would be my guess that if the "primary" database had some sort of failure that lead to a rollback, affected accounts would be locked out until the transactions can be properly recovered from other storage/logging mechanisms.
- jcrites 15y agoA number of techniques help here. These are useful whether or not you're using continuous deployment. A simple approach is not to delete data, but rather simply set a flag which marks the data as deleted. The application storage layer acts as if this data were deleted. A separate process (changed separately) is responsible for actually purging data which has remained deleted for a certain period of time. In any case, do you really wish to delete the data? Keep it around -- it might be useful. The only reason to delete data, outside legal concerns, is to control the cost to retain it. Given that you mostly never really want to delete data, you probably want to orient your schema toward hiding it instead. In that case, if you need to roll back application changes, you can do so by backfilling the data to restore it to a visibile status. Keeping detailed programmatic logs is also helpful for making the backfill easy to execute. For example, in addition to your main textual log file, you keep something like a log consisting of JSON objects, one per line, each which represent the essential details of some application action.
- rmaccloy 15y agoSoft-delete (+ action logs) for data recovery. Also, code review and lots and lots of tests. For schema changes, there are a bunch of approaches. Just a few: * Treat (non-purely additive) schema changes as a special case and take extra care deploying them. (This sounds like a cop-out, but it might be the right effort tradeoff for many startups.) * Write forwards and reverse migrations for each change, and avoid data-losing schema changes. * Implement all (or all non-purely additive) schema changes by shadowing data operations onto a new version of the schema (asynchronously migrate all previous data onto the new schema in some back-end process.) Write tests to verify the migration/shadowed data is correct. Switch data reads to be on the new schema. Eventually delete the old schema in a subsequent deploy after an appropriate stabilization/verification period. I find #3 to be the most general, but there's plenty of overhead (computational, storage and man-hour) that may not be appropriate for your case.
- Pewpewarrows 15y agoThe problem is pretty universally solved: 1. Use some sort of schema migration tool to keep track of and version all changes you've made to the database since your very first commit. 2. Never do a backwards-incompatible schema change. Never. Time and time again these are the ones that will completely bite you in the ass when the day comes that you broke production with a deployment and desperately need the old version back up and running. You're going to end up losing data and shit will hit the fan. It doesn't take much code-wise to support the old scheme, and it will ensure that you have zero downtime while rolling out your updates. 3. Never delete data. The only exception being if some user-submitted content violates a law, like child pornography. In which case, it shouldn't be part of your migration/update process to begin with. Just use deleted flags / boolean fields. Note in your code when certain fields are no longer used anymore because they're only there to support legacy versions.
- jpluscplusm 15y agoI totally get all these points; is there any literature I can point our devs towards to convince them (a) #2 + #3 don't leave any corner cases and (b) they really should take the time to do it (ultimately so I can put an automation solution around the the DB)?
- cdavid 15y agoI am pretty much a beginner in database handling, what would you suggest for 1 ? Concerning 2: how do handle schema fixes (example: at my previous job, the tables were very badly designed, and the application+schema combination prone to frequent race conditions). What to do in this case ? As for 3, I already came to this conclusion on my own, so I was not completely on the wrong track :)
- quicksilver03 15y agoI've used DbDeploy and Mybatis Migrations with good success, they're written in Java but they've also been handy in PHP projects.
- mcfunley 15y agoYou are correct, at Etsy we deploy schema changes once per week. The code always works against both versions of the schema. We never take downtime for schema changes. We avoid data loss by doing soft deletes as much as we can. Sometimes we do want to do real deletes, especially for data that is heavily denormalized, but in those cases we keep an audit trail so that the data can be reconstituted in the event of a mistake.
- macros 15y agoI was at the talk and both of these were addressed. For rollback, the answer was they don't support it, only roll forward. Databases are handled differently from this tool. Iirc he said Thursdays are database day and all changes are done/initiated then.