4 ms·

Real-World Engineering Challenges: Migrations

72 points by 0x54MUR41 4y ago

contingencies 4y ago
This has been recognized for a long time. Those change-over things are really severe. Really severe problems. - Joseph Henry Condon, Bell Labs ... via https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup However, we have much better tools now, such as end to end / black box testing with mature client automation stacks, high availability clustering, CI/CD, distributed architectures, middleware proxies, protocol-level support for temporary failures and retries, mature caching systems and fault tolerant hardware and software infrastructure. So it's practically not as hard as it once was, because if you've got your ducks in a row you can use a battle-proven method and/or get fallbacks for free.
- gonzo41 4y ago
  Yes but all of that is a good idea to implement on a new project starting today with lots of money and a big competent team. Migrations are hard when you have to fight with the project management triangle and then realize that the migration you're doing is on a system which is old, bespoke or not supported. Or all of the above. I like your optimism, but it's always hard work. The most important part of all processes is knowing when you're past a point at which you cannot turn back from.
meerita 4y ago
I remember when we redesigned Fotolog.com to a modern stack it took a huge challenge to write PHP code to migrate trillions of comments, posts, to the new stack written in Javascript. As far I can remember, migrating the photos only (~30TB) took one month, and the data around 2 weeks.
- tommek4077 4y ago
  Is this site gone because of the migration? I get a spam blog on that domain.
  - meerita 4y ago
    The company has been sold, so far I know. I don't know what happen later.
ChrisMarshallNY 4y ago
Migrations are a huge issue, and many companies don't bother. They just wipe the slate clean, dust off the erasers, and start over. I don't have the data (but I'm sure it's there), but I suspect more than one company ended up committing seppuku, over a badly-done migration. I just got finished migrating a fairly crucial SDK, in the app I'm working on. It was one the fundamental "infrastructure" server connectors, and affected everything. It came out great. Far fewer bugs than I expected. In my experience, migrations take planning and strategy. Often, they need to happen over a long time period, in steps. Many companies and engineering teams don't have the patience to manage that.
- dmak 4y ago
  You can't just wipe the slate clean and start over. This is data that needs to be preserved and is a core part of the business. Not even sure how an SDK is related to database migrations?
  - ChrisMarshallNY 4y ago
    It isn't. The title was "migrations." I tend to write my systems in "layers," which makes "migrations" easier. With layers, you can plan a migration in steps. With monoliths, you'd be best advised to pray. We have a "data" migration, coming up; but probably not for a few months. It will be handled in a similar manner. And I have watched "wipe the slate clean" migrations, done, several times. In some cases, they were not deliberate, but the migration screwed up the data so badly, it had to be binned.
    - pc86 4y ago
      > The title was "migrations." Are any of the migrations in the article non-data migrations?
      - ChrisMarshallNY 4y ago
        Fair point. I guess the only "migrations" that matter, are data ones. I'm not a "big data" chap, but I have written distributed (franchised) systems, and have had to do many upgrades and transitions, affecting many folks. It's taught me to be forward-thinking in my architecture.
mooreds 4y ago
Check out GitHub scientist if you are doing a migration with a ruby based system: https://github.com/github/scientist https://github.com/github/scientist Great support and functionality for testing differences between two systems of record.
mooreds 4y ago
Years ago, I was responsible for a small scale migration (thousands of records) and it was really painful. All the edge cases, datatype mismatches, and undocumented functionality really made it a sticky problem. Especially as I was a consultant, billing hourly and trying to provide a reasonable estimate and visibility for the client. Wrote up some more here: https://www.mooreds.com/wordpress/archives/876 https://www.mooreds.com/wordpress/archives/876
kerblang 4y ago
Mostly good stuff, but: > You’ll notice that with backend-only data migrations – like the ones Box or Pinterest executed – there was no need for rollback plans, as the rollout moved forward only after the team had validated that everything worked fine. Speaking from direct experience: That is outright bullshit. Ideally your testing will catch problems, but if you're migrating to a new database architecture, it's an architecture you don't know well, and by the way, your sys admin never even read the installation requirements. This is yet another reason dual-write is ideal, so that you can roll back late in the game when the new DB suddenly seizes up and goes catatonic (esp. Mongo/Cassandra/ElasticSearch). Be that Negative Ned and plan for the worst.
- kerblang 4y ago
  Also: Alpinism is for romantics; engineers seige-climb. Incrementalism is everything. Figure out how to break it down into smaller, safer parts. Big-banging that migration is the almost certain path to failure.
- kerblang 4y ago
  Also-also: Institutional knowledge is bullshit. Old-timers will tell you "Don't worry, the system never needs to do that," and they will be dead wrong. Verify architectural impact at the source.
airbreather 4y ago
I do a lot of control systems migrations for brownfields, where we would be often partially migrating a system both hardware and software. First rule of brownfields cutover - every interface is an opportunity to fail.

72 points by 0x54MUR41 4y ago

This has been recognized for a long time. Those change-over things are really severe. Really severe problems. - Joseph Henry Condon, Bell Labs ... via https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup However, we have much better tools now, such as end to end / black box testing with mature client automation stacks, high availability clustering, CI/CD, distributed architectures, middleware proxies, protocol-level support for temporary failures and retries, mature caching systems and fault tolerant hardware and software infrastructure. So it's practically not as hard as it once was, because if you've got your ducks in a row you can use a battle-proven method and/or get fallbacks for free.

gonzo41 4y ago
Yes but all of that is a good idea to implement on a new project starting today with lots of money and a big competent team. Migrations are hard when you have to fight with the project management triangle and then realize that the migration you're doing is on a system which is old, bespoke or not supported. Or all of the above. I like your optimism, but it's always hard work. The most important part of all processes is knowing when you're past a point at which you cannot turn back from.

meerita 4y ago

I remember when we redesigned Fotolog.com to a modern stack it took a huge challenge to write PHP code to migrate trillions of comments, posts, to the new stack written in Javascript. As far I can remember, migrating the photos only (~30TB) took one month, and the data around 2 weeks.

tommek4077 4y ago
Is this site gone because of the migration? I get a spam blog on that domain.
- meerita 4y ago
  The company has been sold, so far I know. I don't know what happen later.

ChrisMarshallNY 4y ago

Migrations are a huge issue, and many companies don't bother. They just wipe the slate clean, dust off the erasers, and start over. I don't have the data (but I'm sure it's there), but I suspect more than one company ended up committing seppuku, over a badly-done migration. I just got finished migrating a fairly crucial SDK, in the app I'm working on. It was one the fundamental "infrastructure" server connectors, and affected everything. It came out great. Far fewer bugs than I expected. In my experience, migrations take planning and strategy. Often, they need to happen over a long time period, in steps. Many companies and engineering teams don't have the patience to manage that.

dmak 4y ago
You can't just wipe the slate clean and start over. This is data that needs to be preserved and is a core part of the business. Not even sure how an SDK is related to database migrations?
- ChrisMarshallNY 4y ago
  It isn't. The title was "migrations." I tend to write my systems in "layers," which makes "migrations" easier. With layers, you can plan a migration in steps. With monoliths, you'd be best advised to pray. We have a "data" migration, coming up; but probably not for a few months. It will be handled in a similar manner. And I have watched "wipe the slate clean" migrations, done, several times. In some cases, they were not deliberate, but the migration screwed up the data so badly, it had to be binned.
  - pc86 4y ago
    > The title was "migrations." Are any of the migrations in the article non-data migrations?
    - ChrisMarshallNY 4y ago
      Fair point. I guess the only "migrations" that matter, are data ones. I'm not a "big data" chap, but I have written distributed (franchised) systems, and have had to do many upgrades and transitions, affecting many folks. It's taught me to be forward-thinking in my architecture.

mooreds 4y ago

Check out GitHub scientist if you are doing a migration with a ruby based system: https://github.com/github/scientist https://github.com/github/scientist Great support and functionality for testing differences between two systems of record.

mooreds 4y ago

Years ago, I was responsible for a small scale migration (thousands of records) and it was really painful. All the edge cases, datatype mismatches, and undocumented functionality really made it a sticky problem. Especially as I was a consultant, billing hourly and trying to provide a reasonable estimate and visibility for the client. Wrote up some more here: https://www.mooreds.com/wordpress/archives/876 https://www.mooreds.com/wordpress/archives/876

kerblang 4y ago

Mostly good stuff, but: > You’ll notice that with backend-only data migrations – like the ones Box or Pinterest executed – there was no need for rollback plans, as the rollout moved forward only after the team had validated that everything worked fine. Speaking from direct experience: That is outright bullshit. Ideally your testing will catch problems, but if you're migrating to a new database architecture, it's an architecture you don't know well, and by the way, your sys admin never even read the installation requirements. This is yet another reason dual-write is ideal, so that you can roll back late in the game when the new DB suddenly seizes up and goes catatonic (esp. Mongo/Cassandra/ElasticSearch). Be that Negative Ned and plan for the worst.

kerblang 4y ago
Also: Alpinism is for romantics; engineers seige-climb. Incrementalism is everything. Figure out how to break it down into smaller, safer parts. Big-banging that migration is the almost certain path to failure.
kerblang 4y ago
Also-also: Institutional knowledge is bullshit. Old-timers will tell you "Don't worry, the system never needs to do that," and they will be dead wrong. Verify architectural impact at the source.

airbreather 4y ago

I do a lot of control systems migrations for brownfields, where we would be often partially migrating a system both hardware and software. First rule of brownfields cutover - every interface is an opportunity to fail.