6 ms·

> Young dev from large corp: "Sigh, ok, just saying in 2 years from now we wont be in a good spot". Except in my experience it’s not two years, it’s two months

> Young dev from large corp: "Sigh, ok, just saying in 2 years from now we wont be in a good spot". Except in my experience it’s not two years, it’s two months. And I can set up two micro instances with terraform in less than an hour. So soon you’re burning your three months of runway with development at 20-50% efficiency, and have no ability to pivot, if needed, because your infrastructure is rigid and hardcoded. Neither scenario is “correct”, there is only “judgement”, and it can be devastatingly incorrect either way. A lot of people with strong or extreme opinions (too little or too much tech debt) are inadvertently being fooled by “personal survivor bias“.

partomniscient 6y ago
At the same time, it also means collectively we don't know what we're doing to get where we're going and are just throwing stuff at a wall to see what sticks. It also made me think of the Apollo 13 movie. The engineering/process people got the thing into space, the adaptable/dynamic solve it with 'gaffer-tape and whatever is laying around' people got the thing home. We still need both approaches for different reasons. “Personal survivor bias“ accidentally chooses a side for you. Experience eventually makes one realise there aren't diametrically opposing sides, just different capabilities needed in different circumstances.
- p_l 6y ago
  Except the people who did the "tape it together"? Those were the same people who made the long process before. That's why they could just tape it together, they were deep in the knowledge of the system they were fixing and thus could tape a kludge together to get them back.
  - partomniscient 6y ago
    But that's because in those days the people that built the thing have to do production support once it goes live. That doesn't happen so much in larger organisations because they have separate teams divided along "add/build something new" and "keep it going" boundaries without a lot of cross-pollination.
    - p_l 6y ago
      There's also the difference between building bottom up with full knowledge of what you built, like the Apollo program team had (even if they bought something externally, the people who made it were available to the team, and/or the team studied the items in depth). And then there's modern world of slapping together libraries that are often not known, sometimes with closed source, sometimes without real documentation. I've had to use disassembler in the past to access framework we were supposed to use...
kirubakaran 6y ago
I agree. While I'm a big proponent of "as simple as possible", often when people say "x is overkill", they mean "I don't know x" (where x != k8s of course).
- lucasyvas 6y ago
  I think this is 100% true. I've never used the term overkill personally. If I don't know something, I simply say I don't know. And if I do, I always position the trade off of the choice relatively and weigh it.
- OJFord 6y ago
  Do you know k8s?
  - kirubakaran 6y ago
    Haha perhaps not well enough, thus proving my earlier point :-)
- Scarblac 6y ago
  Well introducing some new technology that people will have to learn is often overkill, if there is already a slightly less optimal way to do it that the whole team is familiar with.
- PeterisP 6y ago
  Thing is, it really changes the equation of whether it's worth or not. If you or some other senior engineer knows X, then perhaps they can implement X in an hour and things remain simple. If your team does not know X, then it will take a week to learn X, understand it, try it out on a toy project and implement it in production (which alone might make it an overkill, because it's not worth a week) and it becomes quite plausible that we'll implement it wrong in a way that will bite us later. So unless the benefit is huge, then it's quite reasonable to say "x is overkill" for your team simply because you don't know X.
  - kirubakaran 6y ago
PragmaticPulp 6y ago
> And I can set up two micro instances with terraform in less than an hour. The post was more of a metaphor than a concrete example. However, as a manager I can’t count how many times my team has declared that they can do something in under a day only to have it become a part-time job for someone to maintain indefinitely, or a multi-week rabbit hole as they debug some unexpected behavior, or technical debt as they neglect to document their solution, or a productivity bottleneck as everyone else must now either learn Terraform or wait for the Terraform guy to be available to help and so on. When I transitioned from IC engineer to engineering manager, I quickly learned that engineers frequently assume the best-case scenario for their time estimates, while managers end up planning for the worst-case scenario. The catch is that it’s difficult to tell an engineer that their time estimate could be an order of magnitude or more short of a realistic estimate without implicitly insulting their abilities in the process. ICs also quickly forget that many of us managers were once over-optimistic IC engineers as well.
- eitally 6y ago
  When I was managing a large, distributed team of mostly mediocre engineers (at a poor bigco), it was common to take the best case estimate (provided by a senior engineer or architect), add 50-100% because due to lack of similarly experienced engineers responsible for day-to-day coding, add 25-30% for SQA, and add 50% for "business" delays (usually LOB heads who couldn't agree on features, or who asked for utterly stupid things that we'd have to explain away before we even got started ... like a web UI with a 52 column table meant to emulate the previous Excel-based "system"). The result was frustration all around because the senior engineers knew our estimates were crazy high compared to what they'd be able to do independently, our business stakeholders were mad because of the cost, and our CIO couldn't understand why it seemed literally nothing was doable in less than several thousand person hours. Good times. But when you 1) don't have an constructive relationship between CIO/CTO and business leaders, and 2) offshore everything not to follow-the-sun but purely to save money (and then pay bottom of the barrel even in lower cost regions -- that company still only pays engineers 10-12 lakh in south India), you're going to get an inefficient org creating crap quality software.
  - 6y ago
gowld 6y ago
Twitter had the fail whale for years. Same with reddit. Facebook moved fast and broke things. What engineers often don't understand is that engineering is not important to the business, only the preceptor of it is. Even these huge companies are only backfilling engineering on their core critical components, while new experimental projects are sloppy messes. The only reason half the stuff is solid is that it gets to free ride on comping built for the critical core.
karatestomp 6y ago
> Except in my experience it’s not two years, it’s two months. And I can set up two micro instances with terraform in less than an hour. Fucking that. Only skip the basics entirely if you don't plan to hire anyone in those 5 months (no or inadequate tests, no scripted way to create a dev instance? You just cut their first-two-months productivity in half.) and are 1000% sure no-one's gonna horrifically break prod, and don't mind dev velocity dropping ~10% (and/or bug rate and "of shit it's on fire" calls increasing alarmingly) every month over the next 6 months. There's a lot of room between "we need a k8s cluster with autoscaling, three kinds of databases, lambdas, blah blah" and "LOL develop on prod YOLO!" IME this kind of thing never gets added in month 4 because there's already too much tech debt. Add it in month 1. A couple days and 10s of dollars/month setting up or buying tools & services to keep your team confidently moving fast over the next half-year is some of the most valuable work you'll do. [EDIT] oh and in the specific case of Infra as Code or fully-scripted deployments, around month 4 when your stuff's a bit more complicated it makes your answer "yeah, give me these couple pieces of info and put me in touch with one of their tech guys and I'll have it done by lunch tomorrow" instead of "uhhhh... fuck... uh.... two weeks? Hopefully?" when your sales guy says "we've got a huge deal almost closed but they need our thing hosted on govcloud/on-prem/European servers, can we do that?"
mcv 6y ago
If you can set something up in an hour, don't discuss it; show it. Spend the hour, show it works. If it does, fantastic. If it doesn't, drop it.

partomniscient 6y ago

At the same time, it also means collectively we don't know what we're doing to get where we're going and are just throwing stuff at a wall to see what sticks. It also made me think of the Apollo 13 movie. The engineering/process people got the thing into space, the adaptable/dynamic solve it with 'gaffer-tape and whatever is laying around' people got the thing home. We still need both approaches for different reasons. “Personal survivor bias“ accidentally chooses a side for you. Experience eventually makes one realise there aren't diametrically opposing sides, just different capabilities needed in different circumstances.

p_l 6y ago
Except the people who did the "tape it together"? Those were the same people who made the long process before. That's why they could just tape it together, they were deep in the knowledge of the system they were fixing and thus could tape a kludge together to get them back.
- partomniscient 6y ago
  But that's because in those days the people that built the thing have to do production support once it goes live. That doesn't happen so much in larger organisations because they have separate teams divided along "add/build something new" and "keep it going" boundaries without a lot of cross-pollination.
  - p_l 6y ago
    There's also the difference between building bottom up with full knowledge of what you built, like the Apollo program team had (even if they bought something externally, the people who made it were available to the team, and/or the team studied the items in depth). And then there's modern world of slapping together libraries that are often not known, sometimes with closed source, sometimes without real documentation. I've had to use disassembler in the past to access framework we were supposed to use...

kirubakaran 6y ago

I agree. While I'm a big proponent of "as simple as possible", often when people say "x is overkill", they mean "I don't know x" (where x != k8s of course).

lucasyvas 6y ago
I think this is 100% true. I've never used the term overkill personally. If I don't know something, I simply say I don't know. And if I do, I always position the trade off of the choice relatively and weigh it.
OJFord 6y ago
Do you know k8s?
- kirubakaran 6y ago
  Haha perhaps not well enough, thus proving my earlier point :-)
Scarblac 6y ago
Well introducing some new technology that people will have to learn is often overkill, if there is already a slightly less optimal way to do it that the whole team is familiar with.
PeterisP 6y ago
Thing is, it really changes the equation of whether it's worth or not. If you or some other senior engineer knows X, then perhaps they can implement X in an hour and things remain simple. If your team does not know X, then it will take a week to learn X, understand it, try it out on a toy project and implement it in production (which alone might make it an overkill, because it's not worth a week) and it becomes quite plausible that we'll implement it wrong in a way that will bite us later. So unless the benefit is huge, then it's quite reasonable to say "x is overkill" for your team simply because you don't know X.
- kirubakaran 6y ago

PragmaticPulp 6y ago

> And I can set up two micro instances with terraform in less than an hour. The post was more of a metaphor than a concrete example. However, as a manager I can’t count how many times my team has declared that they can do something in under a day only to have it become a part-time job for someone to maintain indefinitely, or a multi-week rabbit hole as they debug some unexpected behavior, or technical debt as they neglect to document their solution, or a productivity bottleneck as everyone else must now either learn Terraform or wait for the Terraform guy to be available to help and so on. When I transitioned from IC engineer to engineering manager, I quickly learned that engineers frequently assume the best-case scenario for their time estimates, while managers end up planning for the worst-case scenario. The catch is that it’s difficult to tell an engineer that their time estimate could be an order of magnitude or more short of a realistic estimate without implicitly insulting their abilities in the process. ICs also quickly forget that many of us managers were once over-optimistic IC engineers as well.

eitally 6y ago
When I was managing a large, distributed team of mostly mediocre engineers (at a poor bigco), it was common to take the best case estimate (provided by a senior engineer or architect), add 50-100% because due to lack of similarly experienced engineers responsible for day-to-day coding, add 25-30% for SQA, and add 50% for "business" delays (usually LOB heads who couldn't agree on features, or who asked for utterly stupid things that we'd have to explain away before we even got started ... like a web UI with a 52 column table meant to emulate the previous Excel-based "system"). The result was frustration all around because the senior engineers knew our estimates were crazy high compared to what they'd be able to do independently, our business stakeholders were mad because of the cost, and our CIO couldn't understand why it seemed literally nothing was doable in less than several thousand person hours. Good times. But when you 1) don't have an constructive relationship between CIO/CTO and business leaders, and 2) offshore everything not to follow-the-sun but purely to save money (and then pay bottom of the barrel even in lower cost regions -- that company still only pays engineers 10-12 lakh in south India), you're going to get an inefficient org creating crap quality software.
- 6y ago

gowld 6y ago

Twitter had the fail whale for years. Same with reddit. Facebook moved fast and broke things. What engineers often don't understand is that engineering is not important to the business, only the preceptor of it is. Even these huge companies are only backfilling engineering on their core critical components, while new experimental projects are sloppy messes. The only reason half the stuff is solid is that it gets to free ride on comping built for the critical core.

karatestomp 6y ago

> Except in my experience it’s not two years, it’s two months. And I can set up two micro instances with terraform in less than an hour. Fucking that. Only skip the basics entirely if you don't plan to hire anyone in those 5 months (no or inadequate tests, no scripted way to create a dev instance? You just cut their first-two-months productivity in half.) and are 1000% sure no-one's gonna horrifically break prod, and don't mind dev velocity dropping ~10% (and/or bug rate and "of shit it's on fire" calls increasing alarmingly) every month over the next 6 months. There's a lot of room between "we need a k8s cluster with autoscaling, three kinds of databases, lambdas, blah blah" and "LOL develop on prod YOLO!" IME this kind of thing never gets added in month 4 because there's already too much tech debt. Add it in month 1. A couple days and 10s of dollars/month setting up or buying tools & services to keep your team confidently moving fast over the next half-year is some of the most valuable work you'll do. [EDIT] oh and in the specific case of Infra as Code or fully-scripted deployments, around month 4 when your stuff's a bit more complicated it makes your answer "yeah, give me these couple pieces of info and put me in touch with one of their tech guys and I'll have it done by lunch tomorrow" instead of "uhhhh... fuck... uh.... two weeks? Hopefully?" when your sales guy says "we've got a huge deal almost closed but they need our thing hosted on govcloud/on-prem/European servers, can we do that?"

mcv 6y ago

If you can set something up in an hour, don't discuss it; show it. Spend the hour, show it works. If it does, fantastic. If it doesn't, drop it.