3 ms·

Data migrations aren't that hard if you know every system with access to the database. You just need to run the old and the new system in parallel during the mi

Data migrations aren't that hard if you know every system with access to the database. You just need to run the old and the new system in parallel during the migration, ensure you write to both and read from both. Ensure that all tests still pass if you switch source of truth to the new database. At first you treat the old one as the source of truth in production, keep track of how often they disagree, migrate data slowly so you don't overload your database and crash them, after migration disagreements should be 0, then switch source of truth to the new database and remove the migration code, then turn down the old database. Congratulations, you now have migrated data without downtime! There is no special tooling for this since there can't be, you just have to roll up your sleeves and do the dirty work needed. It isn't hard, just a bit tedious if you want to do it properly. A junior developer can do it.

watwut 6y ago
The big issue with data migration are inconsistent data in source, general news in structure and incompatibile approaches in original and destination db structure. The last one may mean you are solving np-complete problems to satisfy new bd.
majormajor 6y ago
"It's not hard, it's just a lot of work." The trick is that the people asking for it almost always assume it's not going to be that much work, and won't be as slow. So politically it's gonna be hard for you if you don't properly set those expectations on day 1.
PeterisP 6y ago
In the tricky data migration cases it's not plausible to know every system and be able to modify them (e.g. to ensure that you write and read from both), many of them will be outside of your control, likely in different organizations. That other organization will not run the same operation on two sets of data to verify if they match - they'll just import it in their workflow, and what happens there will have consequences immediately as soon as some item e.g. a person is switched to a new system. And it's plausible that you won't have any direct visibility in any flaws in that migration until that person complains to that organization and perhaps maybe eventually they will identify that your data migration is involved instead of simply screwing that person for months.

watwut 6y ago

The big issue with data migration are inconsistent data in source, general news in structure and incompatibile approaches in original and destination db structure. The last one may mean you are solving np-complete problems to satisfy new bd.

majormajor 6y ago

"It's not hard, it's just a lot of work." The trick is that the people asking for it almost always assume it's not going to be that much work, and won't be as slow. So politically it's gonna be hard for you if you don't properly set those expectations on day 1.

PeterisP 6y ago

In the tricky data migration cases it's not plausible to know every system and be able to modify them (e.g. to ensure that you write and read from both), many of them will be outside of your control, likely in different organizations. That other organization will not run the same operation on two sets of data to verify if they match - they'll just import it in their workflow, and what happens there will have consequences immediately as soon as some item e.g. a person is switched to a new system. And it's plausible that you won't have any direct visibility in any flaws in that migration until that person complains to that organization and perhaps maybe eventually they will identify that your data migration is involved instead of simply screwing that person for months.