4 ms·
The takeway IMO is that the reason their transition was so seamless is because they took the time to do it 'the right way' - that is: set up the new system in p
by Jeema101 6y ago
The takeway IMO is that the reason their transition was so seamless is because they took the time to do it 'the right way' - that is: set up the new system in parallel, duplicating the reads and writes, and verifying that things looked good every step of the way.
That gives you not only a long window to test and validate, but also the option to rollback if things suddenly don't look so good with the new system. It seems like people sometimes get impatient and want to do a 'big switchover' with insufficient production testing and no real rollback plan, but in my experience that's almost never a good idea.
- lma21 6y agoIndeed, though that must have cost them twice as much to do the migration with the two systems being live in parallel. Some choose to avoid the extra cost.
- tyre 6y agoCost is not only measured in dollars for servers. The cost of doing a massive, irreversible migration that doesn’t work seamlessly the first time is significant. And often more significant in engineering time, lost revenue from services going offline, trust both from external users and internal teams watching the shitshow, and possibly data loss or backwards-incompatible changes than those server dollars. It’s funny that one consistent theme of the best engineers I’ve worked with is less their “amazing 10x ideas” than the number of “sounds like a good idea if you’ve never done it” ideas that they shut down. Huge migrations are one.
- acdha 6y agoI had an example of that recently: we’ve been buying grocery boxes from Hungry Harvest. They announced a new website was coming a couple of weeks ago and had the usually puffery about how it’d be better in vague ways (the old one was not noticeably bad). Then the site was down longer than expected. Then everyone was told their order customizations were lost. Now it’s so bad that they’re skipping an entire week’s delivery for all of their customers. I don’t know how they got there but that’s expensive, disrupts their supply chain, and I suspect they’d cheerfully pay four times as much not to have to slog through this mess.
- alephu5 6y agoThey're running their own servers so these databases probably cost fractions of a cent per month.
- tinus_hn 6y agoActually if you are not running your own servers you can just pay double for a month instead of buying the hardware and having it surplus when the migration is done.