4 ms·
Ugh. I hear this all the time. While it's somewhat correct during early stages, it can be very wrong soon afterwards. I left a company after an extremely short
by raffraffraff 4y ago
Ugh. I hear this all the time. While it's somewhat correct during early stages, it can be very wrong soon afterwards. I left a company after an extremely short time because it turned out they spent a decade ignoring scalability. They ended up completely boxed in and had no clue how to fix it. After a decade of piling things on top of each other it's extremely difficult to fix without throwing large pieces out, but they didn't want to do that. We were expected to refactor the pancake stack while simultaneously building new data centers to the old design (because being able to deploy to more than one region isn't just about "SCALABILITY DUH" it's also about data laws, regulatory requirements etc and therefore selling to certain customers). And none of the infra code was written to scale, it was pretty much hard-coded to the original data center and required thousands of search-and-replace that weren't as simple as a 'sed' operation because of assumptions like "all regions have 3+ availability zones" or failure to consider globally unique naming.
The other problem with continually and habitually silencing (perfectly valid) scaling concerns is that the people who are being condescendingly told that they're "prematurely optimising" will eventually leave. When you turn this article's advice into a Harry Potter-esque spell that somebody can shout during a design session: "Premature Optimisation!" it will get abused by people who don't like detail, or want to make some stupid deadline that they agreed to before they thought deeply enough about the problem space. How long can you retain good engineers if they're constantly battling against this? When you finally saturate some choke point and your system begins to fail in unexpected ways, or your costs are too high and you suddenly realise that scalability also means scaling down too, those people who you silenced might not be there.
I was once told, while cobbling together a metrics solution spanning 8 data centers that some of the things I wanted to spend a day or two on were "premature optimisation". Until we switched it on. It was dropping 50% of metrics on the floor (useless for alerting) and we got a call from the networks team because it was saturating the links. I spend those two days on my optimisations it worked. We were extremely lucky that the breakage happened at the start, and not months later. We were lucky that it didn't just turn into a random alert for a random team in the middle of the night. We were lucky that we had a network team monitoring things adequately. We were lucky that we didn't build more things on top of the bad design that relied on it.
Sometimes one man's premature optimisation is another man's due-diligence. You need to be sure that your rewarding the right behaviours.
- P5fRxh5kUvp2th 4y agoright/wrong isn't the best mindset here. Code that sits in production working successfully for 6 months only to be replaced is still good code. If something works for decades and then has to be replaced due to scale, that's a good problem to have. The unwillingness to replace things is where it all goes awry. OTOH, wanting to _rewrite_ should almost always be told no.
- raffraffraff 4y agoI completely agree. But I also think that these problems never emerge in a big-bang way. By year two, or engineering hire #30, or some other heuristic, you should probably be replacing some of that stuff, refactoring other stuff, and certainly thinking about refactoring teams or hiring to fill specific functions that didn't need to exist at the start. There were so many red flags at that company, and inability to scale was just one, for sure. But these things sometimes come as a complete package. I think my problem with the article is that it thinks it's selling some generically applicable advice. But the wrong type of company with inexperienced engineers can take the wrong message from that article and totally fuck up an otherwise promising product.