3 ms·

The paradoxical thing about this for me is that both the most junior and senior engineers fall victim to this line of thinking. I have done so myself, on both

The paradoxical thing about this for me is that both the most junior and senior engineers fall victim to this line of thinking. I have done so myself, on both ends of the experience/skill spectrum. As a junior it was because the tech was exciting, and the cool kids were doing it so obviously it was the Right Way to do things. As a senior because I'd been a part of the scramble to scale short-sighted systems in a startup that had found product/market fit and didn't want that pain again. Turns out you can build the most scalable thing and people still won't flock to your product. And you wasted all that iteration and learning time.

lbriner 5y ago
This is true! What I have learned is only to plan for the next order of magnitude from a sensible starting point. If you are creating a new e.g. AirBnB, you will need a minimim viable performance, let's say 1000 users at a time on the site. After that, you plan for 10000, then 100000 etc. You might get multiple orders of magnitude from the same improvements or changes but you only need to get one more step. Why? Your team and expertise will be different in 3 years. You might get bought-out or a new thing might come along that suits your model really well. Eventually, you might have enough money to have an entire data centre with 100 support staff but you certainly can't plan for that on day one.
- twistedpair 5y ago
  > You might get bought-out or a new thing might come along that suits your model really well. Once you get bought out, you'll likely need to integrate or replatform to the buyer's tech. Just look @ Google acquisitions. They spend the next few years paying down tech debt and scaling, so don't waste your time on that pre liquidity event. Spend that period of time on product improvements and market expansion.
- jconley 5y ago
  I do the same. But I've done it enough times now that I usually start at the 10k number, just to survive a huge press surge without trouble. Hardware and software is good enough now that this isn't usually much of an overhead.
- hintymad 5y ago
  > What I have learned is only to plan for the next order of magnitude from a sensible starting point. This is exactly what Google suggested. In particular, it was Jeff Dean who shared this piece of advice in one of his talks on scaling Google's infrastructure.
bcrosby95 5y ago
My friends and I call this the "complexity hump". Some people never get over it.
- hinkley 5y ago
  I have a growing pile of anecdotes about smart people getting trapped in this hump, possibly confirmation bias based on recent experience. But it has made me revise my esteem for former coworkers who were very smart and avoided the trap.
fibers 5y ago
Isn't this a symptom of insanely crazy programming whiteboarding interviews?
- xboxnolifes 5y ago
  Why do you think that is the case?
- jconley 5y ago
  I refuse to do those and have fallen victim to it as a founder, so not in my case.
hinkley 5y ago
I have spent more time in my career than many people think is healthy trying to plot out contingency plans for 'what if' situations. It stops being irritating the moment the building is on fire and all of a sudden people want to listen to me. My relationship with YAGNI has ebbed and flowed over the years, and as with any 'problematic' relationship, you may or may not see your own relationship clearly but people who have it worse are still easy to identify. Whether you then ask if you're like that is a matter of wisdom. What we hear about, what we interview about, what we write about is these cool huge projects from the heroes we want, but the heroes we need are the people who figure out how to solve problems in ways that leave the option of solving them differently in the future. It's a little dangerous to say things like that out loud because lots of people hear that as "I'm going to build a configuration engine that I can use to swap out implementations at startup/runtime," which is pretty much the merry-go-round we are all stuck on half the time. No, what I mean is go back way old school, taking some notes from Bertrand Meyer, and arranging your code so that there are 'spots' where major changes in functionality seem to naturally fit. I may have mangled this story in my head over the years into a parable, but I still recall hearing my uncles waxing poetic about, the Chevy straight 6 engine block that was popular during the height of the muscle car era. This engine did not have a particularly high horse power to displacement ratio. But in those days engine bays were fairly empty, and the Straight 6 was overbuilt just enough that it was a dream to modify it, and easy enough to work on that many people did. They bored it out for higher displacement, modified it for higher compression ratios (naturally aspirated or blown), hung additional accessories off of it, you name it. Many people were running around with cars that had over 50% more horse power than the stock version, and some crazy bastards who went considerably higher. In short, this engine was not that particularly great, but it was full of potential. As someone who may or may not become the next Google, (it's been a long time since I worked for anyone who shared opinions like that, when Google was smaller and only dreamed of being as big as they are now), I don't want a system full of features. I want a system full of possibilities. IF we wanted to do this, you would add it here, and verify it here. But we don't, not yet, so if you could just not break 'here' please, that would be great. Nobody is collecting those stories and patterns. They're hidden away in Meyer, Fowler, dropped as throw-away lines in lectures, and discussed at length over coffee, noodles, or beers but never written down. Elsewhere they are really hidden in older aphorisms like 'premature optimization', "single responsibility" and that ilk, so much so that they risk becoming bad advice.

lbriner 5y ago

This is true! What I have learned is only to plan for the next order of magnitude from a sensible starting point. If you are creating a new e.g. AirBnB, you will need a minimim viable performance, let's say 1000 users at a time on the site. After that, you plan for 10000, then 100000 etc. You might get multiple orders of magnitude from the same improvements or changes but you only need to get one more step. Why? Your team and expertise will be different in 3 years. You might get bought-out or a new thing might come along that suits your model really well. Eventually, you might have enough money to have an entire data centre with 100 support staff but you certainly can't plan for that on day one.

twistedpair 5y ago
> You might get bought-out or a new thing might come along that suits your model really well. Once you get bought out, you'll likely need to integrate or replatform to the buyer's tech. Just look @ Google acquisitions. They spend the next few years paying down tech debt and scaling, so don't waste your time on that pre liquidity event. Spend that period of time on product improvements and market expansion.
jconley 5y ago
I do the same. But I've done it enough times now that I usually start at the 10k number, just to survive a huge press surge without trouble. Hardware and software is good enough now that this isn't usually much of an overhead.
hintymad 5y ago
> What I have learned is only to plan for the next order of magnitude from a sensible starting point. This is exactly what Google suggested. In particular, it was Jeff Dean who shared this piece of advice in one of his talks on scaling Google's infrastructure.

bcrosby95 5y ago

My friends and I call this the "complexity hump". Some people never get over it.

hinkley 5y ago
I have a growing pile of anecdotes about smart people getting trapped in this hump, possibly confirmation bias based on recent experience. But it has made me revise my esteem for former coworkers who were very smart and avoided the trap.

fibers 5y ago

Isn't this a symptom of insanely crazy programming whiteboarding interviews?

xboxnolifes 5y ago
Why do you think that is the case?
jconley 5y ago
I refuse to do those and have fallen victim to it as a founder, so not in my case.

hinkley 5y ago

I have spent more time in my career than many people think is healthy trying to plot out contingency plans for 'what if' situations. It stops being irritating the moment the building is on fire and all of a sudden people want to listen to me. My relationship with YAGNI has ebbed and flowed over the years, and as with any 'problematic' relationship, you may or may not see your own relationship clearly but people who have it worse are still easy to identify. Whether you then ask if you're like that is a matter of wisdom. What we hear about, what we interview about, what we write about is these cool huge projects from the heroes we want, but the heroes we need are the people who figure out how to solve problems in ways that leave the option of solving them differently in the future. It's a little dangerous to say things like that out loud because lots of people hear that as "I'm going to build a configuration engine that I can use to swap out implementations at startup/runtime," which is pretty much the merry-go-round we are all stuck on half the time. No, what I mean is go back way old school, taking some notes from Bertrand Meyer, and arranging your code so that there are 'spots' where major changes in functionality seem to naturally fit. I may have mangled this story in my head over the years into a parable, but I still recall hearing my uncles waxing poetic about, the Chevy straight 6 engine block that was popular during the height of the muscle car era. This engine did not have a particularly high horse power to displacement ratio. But in those days engine bays were fairly empty, and the Straight 6 was overbuilt just enough that it was a dream to modify it, and easy enough to work on that many people did. They bored it out for higher displacement, modified it for higher compression ratios (naturally aspirated or blown), hung additional accessories off of it, you name it. Many people were running around with cars that had over 50% more horse power than the stock version, and some crazy bastards who went considerably higher. In short, this engine was not that particularly great, but it was full of potential. As someone who may or may not become the next Google, (it's been a long time since I worked for anyone who shared opinions like that, when Google was smaller and only dreamed of being as big as they are now), I don't want a system full of features. I want a system full of possibilities. IF we wanted to do this, you would add it here, and verify it here. But we don't, not yet, so if you could just not break 'here' please, that would be great. Nobody is collecting those stories and patterns. They're hidden away in Meyer, Fowler, dropped as throw-away lines in lectures, and discussed at length over coffee, noodles, or beers but never written down. Elsewhere they are really hidden in older aphorisms like 'premature optimization', "single responsibility" and that ilk, so much so that they risk becoming bad advice.