3 ms·

Is that supposed to make it seem better? If refactoring is too hard for a Microsoft owned company what am I to think about my tech stack?

by sqeaky 3y ago

bostik 3y ago
Beyond ludicrously small systems, refactoring of live 24/7 production systems is never easy. Reality has a surprising amount of detail, and any non-trivial, customer-facing system will have accumulated weird code paths to account for obscure but nonetheless expensive edge cases. A codebase built across >20 years, scaled to support millions of concurrent users is going to be absolutely filled to the brim with weird things. When you add the need for live migrations with zero downtime, done every few years to account for next order of magnitude loads, you end up with a proper Frankenstein's monster. It's not called "rebuilding an airplane while flying" for a lark. Every round includes a long, complex engineering effort of incremental live migration. With parallel read/write patterns between old and new systems, and all their annoying semantic differences. And then, to add insult to injury, while your core team was going through the months-long process of migrating one essential service, half a dozen upstream teams have independently realised they can depend on some weird side effects of the intermediate state and embedded its assumptions to a Critical Business Process[tm] responsible for a decent fraction of your company's monthly revenue. Breaking their implicit workflow will make your entire company go from black to red, so your core team is now saddled with supporting the known-broken assumptions. Then you get to add wildly differing latency profiles to the mix. While you were running on your own hardware, the worst-case latency was rack-to-rack. Implicit assumptions on massive but essential workloads may depend, unknowingly, on call-to-call latencies that only rarely exceed 100 microseconds. In a cross-AZ cloud setting you may suddenly have a p90 floor of 0.2ms. A lot of software can break in unexpected ways when things are consistently just a little bit too slow. Welcome to the wonderful world of distributed systems and cloud migrations. At some point the scars will heal. Allegedly.

Is that supposed to make it seem better? If refactoring is too hard for a Microsoft owned company what am I to think about my tech stack?

by sqeaky 3y ago

Is that supposed to make it seem better? If refactoring is too hard for a Microsoft owned company what am I to think about my tech stack?

bostik 3y ago

Beyond ludicrously small systems, refactoring of live 24/7 production systems is never easy. Reality has a surprising amount of detail, and any non-trivial, customer-facing system will have accumulated weird code paths to account for obscure but nonetheless expensive edge cases. A codebase built across >20 years, scaled to support millions of concurrent users is going to be absolutely filled to the brim with weird things. When you add the need for live migrations with zero downtime, done every few years to account for next order of magnitude loads, you end up with a proper Frankenstein's monster. It's not called "rebuilding an airplane while flying" for a lark. Every round includes a long, complex engineering effort of incremental live migration. With parallel read/write patterns between old and new systems, and all their annoying semantic differences. And then, to add insult to injury, while your core team was going through the months-long process of migrating one essential service, half a dozen upstream teams have independently realised they can depend on some weird side effects of the intermediate state and embedded its assumptions to a Critical Business Process[tm] responsible for a decent fraction of your company's monthly revenue. Breaking their implicit workflow will make your entire company go from black to red, so your core team is now saddled with supporting the known-broken assumptions. Then you get to add wildly differing latency profiles to the mix. While you were running on your own hardware, the worst-case latency was rack-to-rack. Implicit assumptions on massive but essential workloads may depend, unknowingly, on call-to-call latencies that only rarely exceed 100 microseconds. In a cross-AZ cloud setting you may suddenly have a p90 floor of 0.2ms. A lot of software can break in unexpected ways when things are consistently just a little bit too slow. Welcome to the wonderful world of distributed systems and cloud migrations. At some point the scars will heal. Allegedly.