5 ms·

I recently (mostly) finished a legacy data migration and here's my advice: Treat it like you would treat a climbing expedition on one of the world's dangerous p

I recently (mostly) finished a legacy data migration and here's my advice: Treat it like you would treat a climbing expedition on one of the world's dangerous peaks. It's a given that failure is more likely than success; in fact it's potentially worst-case-scenario catastrophic. You will encounter many optimists along the way, and these are your most dangerous enemies - get them off your team ASAP. The people you can trust are the pessimists who insist on double-checking every assumption (how old is that rope?) and worrying about the what-if's. If folks can't accept your "attitude problem", walk away.

0xFFFE 6y ago
I wish I could up vote this twice. Assume every step in the process will fail and have contingencies. Don't set yourself unrealistic timelines, this will create pressure on you which will lead to more mistakes.
- hinkley 6y ago
  We worked on a Lean Process system within a larger Waterfall organization, and for the first year we constantly got called out for announcing that some milestone was going to be two weeks late. See? You're doing it wrong, you're doing it wrong. After a while we noticed a pattern. Whoever 'blinked' first got a bunch of theatrics, including claiming a week for week slip every time (even if the problem was only one feature they barely used). Eventually half the teams would also claim a week slip. Anyone who missed that second deadline, the same process would repeat. Some teams who weren't 'the bottleneck' were still writing code week three. Eventually we figured out that when we said a week, we meant 5-6 business days. When everyone else said a week, they meant 8-15 business days, and we suspected corner cutting even in those numbers. Over time we started 'helping out' people at the interfaces, slowly taking over ownership of more and more surface area. And yet these other teams still struggled to ship on time. By the end I realized that this was the organizational principle the company was built on. Take a 2d plane of responsibilities, distribute people randomly on the plane, and where one fails to thrive, its neighbors grow to fill the space. You ship when most of the map is covered, and the gaps represent emergencies and/or lawsuits.
hinkley 6y ago
Are you hiring? Asking for a friend.
- kerblang 6y ago
  Nope, sorry
wiz21c 6y ago
Basically when you migrate a project from A to B, your best result would be to have B that works as good as A (people don't want change, people don't want new bugs). Since you most of the time didn't code A, were not responsible for it, etc. and, at the same time, are expected to understand it thoroughly (you got the fg code after all), you're expected to do at least as good as A. So you're B will be compared to A and will always be seen as inferior. How treacherous.
username90 6y ago
Data migrations aren't that hard if you know every system with access to the database. You just need to run the old and the new system in parallel during the migration, ensure you write to both and read from both. Ensure that all tests still pass if you switch source of truth to the new database. At first you treat the old one as the source of truth in production, keep track of how often they disagree, migrate data slowly so you don't overload your database and crash them, after migration disagreements should be 0, then switch source of truth to the new database and remove the migration code, then turn down the old database. Congratulations, you now have migrated data without downtime! There is no special tooling for this since there can't be, you just have to roll up your sleeves and do the dirty work needed. It isn't hard, just a bit tedious if you want to do it properly. A junior developer can do it.
- watwut 6y ago
  The big issue with data migration are inconsistent data in source, general news in structure and incompatibile approaches in original and destination db structure. The last one may mean you are solving np-complete problems to satisfy new bd.
- majormajor 6y ago
  "It's not hard, it's just a lot of work." The trick is that the people asking for it almost always assume it's not going to be that much work, and won't be as slow. So politically it's gonna be hard for you if you don't properly set those expectations on day 1.
- PeterisP 6y ago
  In the tricky data migration cases it's not plausible to know every system and be able to modify them (e.g. to ensure that you write and read from both), many of them will be outside of your control, likely in different organizations. That other organization will not run the same operation on two sets of data to verify if they match - they'll just import it in their workflow, and what happens there will have consequences immediately as soon as some item e.g. a person is switched to a new system. And it's plausible that you won't have any direct visibility in any flaws in that migration until that person complains to that organization and perhaps maybe eventually they will identify that your data migration is involved instead of simply screwing that person for months.
OldHand2018 6y ago
My organization narrowly avoided a business-crushing event a few months ago. Years of optimistic promises were finally put to rest and the sobering reality came to light: our legacy migration is, at best, going to be done in 3-5 years. We were just barely within the window to order a pair of new servers before the Itanium order book closed forever. Naturally, our old servers are too old for VSI to support, so HP was willing to sell us a license to give us time to migrate from the old legacy servers to the new legacy servers. It came with a warning that THIS IS THE LAST OPENVMS LICENSE YOU WILL EVER GET FROM HP AND YOUR CEO MUST SIGN THIS BEFORE WE GIVE IT TO YOU.
- pabs3 6y ago
  Its a shame FreeVMS never took off: http://www.pvv.org/~roart/freevms.html http://www.pvv.org/~roart/freevms.html https://github.com/rroart/freevms https://github.com/rroart/freevms
mathattack 6y ago
My fear is that the “skeptics” who really know what’s going on are rapidly retiring. Nobody can even patch these systems.

0xFFFE 6y ago

I wish I could up vote this twice. Assume every step in the process will fail and have contingencies. Don't set yourself unrealistic timelines, this will create pressure on you which will lead to more mistakes.

hinkley 6y ago
We worked on a Lean Process system within a larger Waterfall organization, and for the first year we constantly got called out for announcing that some milestone was going to be two weeks late. See? You're doing it wrong, you're doing it wrong. After a while we noticed a pattern. Whoever 'blinked' first got a bunch of theatrics, including claiming a week for week slip every time (even if the problem was only one feature they barely used). Eventually half the teams would also claim a week slip. Anyone who missed that second deadline, the same process would repeat. Some teams who weren't 'the bottleneck' were still writing code week three. Eventually we figured out that when we said a week, we meant 5-6 business days. When everyone else said a week, they meant 8-15 business days, and we suspected corner cutting even in those numbers. Over time we started 'helping out' people at the interfaces, slowly taking over ownership of more and more surface area. And yet these other teams still struggled to ship on time. By the end I realized that this was the organizational principle the company was built on. Take a 2d plane of responsibilities, distribute people randomly on the plane, and where one fails to thrive, its neighbors grow to fill the space. You ship when most of the map is covered, and the gaps represent emergencies and/or lawsuits.

hinkley 6y ago

Are you hiring? Asking for a friend.

kerblang 6y ago
Nope, sorry

wiz21c 6y ago

Basically when you migrate a project from A to B, your best result would be to have B that works as good as A (people don't want change, people don't want new bugs). Since you most of the time didn't code A, were not responsible for it, etc. and, at the same time, are expected to understand it thoroughly (you got the fg code after all), you're expected to do at least as good as A. So you're B will be compared to A and will always be seen as inferior. How treacherous.

username90 6y ago

Data migrations aren't that hard if you know every system with access to the database. You just need to run the old and the new system in parallel during the migration, ensure you write to both and read from both. Ensure that all tests still pass if you switch source of truth to the new database. At first you treat the old one as the source of truth in production, keep track of how often they disagree, migrate data slowly so you don't overload your database and crash them, after migration disagreements should be 0, then switch source of truth to the new database and remove the migration code, then turn down the old database. Congratulations, you now have migrated data without downtime! There is no special tooling for this since there can't be, you just have to roll up your sleeves and do the dirty work needed. It isn't hard, just a bit tedious if you want to do it properly. A junior developer can do it.

watwut 6y ago
The big issue with data migration are inconsistent data in source, general news in structure and incompatibile approaches in original and destination db structure. The last one may mean you are solving np-complete problems to satisfy new bd.
majormajor 6y ago
"It's not hard, it's just a lot of work." The trick is that the people asking for it almost always assume it's not going to be that much work, and won't be as slow. So politically it's gonna be hard for you if you don't properly set those expectations on day 1.
PeterisP 6y ago
In the tricky data migration cases it's not plausible to know every system and be able to modify them (e.g. to ensure that you write and read from both), many of them will be outside of your control, likely in different organizations. That other organization will not run the same operation on two sets of data to verify if they match - they'll just import it in their workflow, and what happens there will have consequences immediately as soon as some item e.g. a person is switched to a new system. And it's plausible that you won't have any direct visibility in any flaws in that migration until that person complains to that organization and perhaps maybe eventually they will identify that your data migration is involved instead of simply screwing that person for months.

OldHand2018 6y ago

My organization narrowly avoided a business-crushing event a few months ago. Years of optimistic promises were finally put to rest and the sobering reality came to light: our legacy migration is, at best, going to be done in 3-5 years. We were just barely within the window to order a pair of new servers before the Itanium order book closed forever. Naturally, our old servers are too old for VSI to support, so HP was willing to sell us a license to give us time to migrate from the old legacy servers to the new legacy servers. It came with a warning that THIS IS THE LAST OPENVMS LICENSE YOU WILL EVER GET FROM HP AND YOUR CEO MUST SIGN THIS BEFORE WE GIVE IT TO YOU.

pabs3 6y ago
Its a shame FreeVMS never took off: http://www.pvv.org/~roart/freevms.html http://www.pvv.org/~roart/freevms.html https://github.com/rroart/freevms https://github.com/rroart/freevms

mathattack 6y ago

My fear is that the “skeptics” who really know what’s going on are rapidly retiring. Nobody can even patch these systems.