5 ms·

Not building what doesn't need to exist won't get you promoted. You don't get promoted by avoiding entire classes of problems with a simple, reliable system. Yo

Not building what doesn't need to exist won't get you promoted. You don't get promoted by avoiding entire classes of problems with a simple, reliable system. You get promoted by pulling heroics to build some absurdly complex cloud Kubernetes bullshit, and then pulling more heroics to fix the endless stream of production issues that will result from your overengineered nightmare.

Jtsummers 5y ago
http://web.mit.edu/nelsonr/www/Repenning%3DSterman_CMR_su01_.pdf http://web.mit.edu/nelsonr/www/Repenning%3DSterman_CMR_su01_... I've posted this here before, though got no comments. But yes, firefighter/arsonists get more promotions and kudos than careful thinkers who don't bother setting fires to fight. EDIT: One past discussion here (2015): https://news.ycombinator.com/item?id=8940820 https://news.ycombinator.com/item?id=8940820
- Aqueous 5y ago
  It is very bad. I spent the last year or so rebuilding a system with a fundamental architecture error that it would likely be fighting against for decades. My reward? The refactor was so difficult to do that I ended up getting dinged on timeline. And promotions got delayed The incentive structures at big companies are out of whack. Instead of the original developer, who was promoted to staff engineer, gettting dinged for making the mistake, I got dinged for not fixing it fast enough, and my career slowed down as a result. The refactor was incredibly difficult and launched without issue and I am probably back on track for a promotion, but I have started to make noise about the incentives being askew
  - Buttons840 5y ago
    > I have started to make noise about the incentives being askew Choosing the hard path again I see.
    - Aqueous 5y ago
      Haha, yep. I just think it's fundamentally unjust for the developers doing the (easiest) green field development to get promoted while making significant architecture mistakes, then punishing later developers for pointing out those mistakes and getting stuck with fixing them. Until someone makes noise, this is just going to keep happening.
  - octopoc 5y ago
    It's a big red flag when a dev makes something really cool that is (according to them) 90% done, then hands it off to someone else to finish. It's a sign that they: 1. Can't finish what they started 2. Are going to rewrite stuff unnecessarily 3. Are willing to throw their coworkers under the bus
    - Aqueous 5y ago
      I will say, in this case it is not what happened. The previous system launched successfully with an incorrect schema. The platform grew and when it came time to add more features, some of them were either impossible to do because of the incorrect schema, or they would have been so complex to implement that the system would have fallen apart. Needless to say, the schema needed to be corrected, but the wrong schema was already baked in all across the platform. So it's not like the main developer handed something off that was unfinished. They handed something off that was finished but wrong. And because of that we had to replace the engine while in flight, so to speak.
  - shepherdjerred 5y ago
    I've had this exact same experience. Half-baked, unmaintainable features are launched. The developers get promoted and leave the team. The team is left to clean up and pay the consequences until development speed slows to a crawl. The team isn't able to get promoted because they're busy doing boring, undervalued work. The engineers responsible for the mess ultimately end up ahead. If your goal is to get promoted then clearly the best path is to take shortcuts and not act in the long-term interest of the team. You'll get promoted for it. If your goal is to build a product that won't be burdened in operations/maintenance then you have to move slowly, correctly, and you'll be setting yourself back. This was my experience at a FANG.
    - Aqueous 5y ago
      Precisely. If I were to have neglected to point out the issue, or refused to lead the fix, I would probably have been promoted by now. Instead I chose the hard path and got punished for it. However, the team will benefit far into the future because they will no longer be fighting a schema that was just wrong. The only cost they had to pay was my career velocity.
  - gajjanag 5y ago
    And companies wonder why loyalty is hard to find... Far easier to jump ship at opportune moments for fast tracking a career than fighting dysfunctional internal incentive structures.
papito 5y ago
"Complexity kills" is an old mantra a lot of us fossils are familiar with, but the new kids still need to learn this lesson the hard way. What makes things worse is the ability to stitch together twenty different AWS services together because - well - you can. Solving hard problems in simple and maintainable ways, I would argue, IS the job. But what I see is engineers solving problems they don't even have, for THIS IS THE WAY. Also, there is a FAANG guy on the team that claims that's how they did it over there.
- scruple 5y ago
  The cargo cult mentality is the thing that I've seen throughout my career. FAANG does it this way therefore we must do it this way (I've also heard it parroted at the senior leadership and executive levels more than once!). IMO, this is the driving force behind juniors and intermediates who pull too much complexity into their solutions.
  - deckard1 5y ago
    Yes, and I've commented about this again and again. The push for microservices, K8S, Docker. It's all the same thing. It all comes from Google. Or occasionally Facebook. We can thank them for many of the bad ideas of the past 20 years. Open floorplan offices. K8S could only have come from Google. It's a reflection of their corporate structure. Conway's Law and all that. Much like microservices. It's a reflection of their internal chaos. And yet every company that isn't Google tries to emulate them and fails. Then they wonder why microservices suck so much. It's not that microservices suck. It's that Google sucks. Google sucks so much that microservices are pretty much the only way they can operate as an entity.
    - scruple 5y ago
      I was going to type out something about the lack of cohesive technical vision being a massive part of the problem but Conway's Law is far too succinct.
    - deleted 5y ago
      [deleted]
jtdev 5y ago
Technical employees who frequently act like firefighters (especially those that draw attention to themselves when doing so, i.e., the martyr types) are toxic and should be removed ASAP.
- rixed 5y ago
  No more firefighters, no more fires! Qed.
flatline 5y ago
Is k8s really considered that complex? Yes, you need a lot of tooling to go from code to orchestration. But when you are commoditizing software to run on a desktop, on prem, in the cloud, across many customers and topologies, reproducible deployments are way simpler than hand-rolled whatevers.
- logshipper 5y ago
  From what I understand, the labour/complexity in K8S lies not with the deployment, but with managing said deployment over the long term. This is a real problem for system architects/security professionals that need to maintain a bird's eye view of the system - a job that gets increasingly complex with every additional microservice once you hit critical mass. This is not to say that K8S does not work - they do, but for certain problems. The cargo-cult mentality, however, threatens to introduce complexity in systems where there is need for none.
- paulddraper 5y ago
  Not exactly. But k8s enables a lot of complexity that is usually underestimated.
- henning 5y ago
  Kubernetes itself is not the problem. It's the DIY PaaS setups people build with YAML files, bash scripts, Jenkins jobs and webhooks that people build on top of it.
paulddraper 5y ago
I swear 50% of microservices is feeling really productive as you have invented so many more things that need gluing together.
ahallock 5y ago
Failing upward and job security from complex systems is a strange thing. It's never done outright. Like, I've never had a conversation with another engineer where we conspire or they tell me they're going to or have built a complex system just for this purpose. It's only observed amusingly after the fact.
beebmam 5y ago
People need to stop trying to get promoted then. I'm very happy building simple and easy to maintain systems without flashy features. I couldn't care less about a promotion.

Jtsummers 5y ago

http://web.mit.edu/nelsonr/www/Repenning%3DSterman_CMR_su01_.pdf http://web.mit.edu/nelsonr/www/Repenning%3DSterman_CMR_su01_... I've posted this here before, though got no comments. But yes, firefighter/arsonists get more promotions and kudos than careful thinkers who don't bother setting fires to fight. EDIT: One past discussion here (2015): https://news.ycombinator.com/item?id=8940820 https://news.ycombinator.com/item?id=8940820

Aqueous 5y ago
It is very bad. I spent the last year or so rebuilding a system with a fundamental architecture error that it would likely be fighting against for decades. My reward? The refactor was so difficult to do that I ended up getting dinged on timeline. And promotions got delayed The incentive structures at big companies are out of whack. Instead of the original developer, who was promoted to staff engineer, gettting dinged for making the mistake, I got dinged for not fixing it fast enough, and my career slowed down as a result. The refactor was incredibly difficult and launched without issue and I am probably back on track for a promotion, but I have started to make noise about the incentives being askew
- Buttons840 5y ago
  > I have started to make noise about the incentives being askew Choosing the hard path again I see.
  - Aqueous 5y ago
    Haha, yep. I just think it's fundamentally unjust for the developers doing the (easiest) green field development to get promoted while making significant architecture mistakes, then punishing later developers for pointing out those mistakes and getting stuck with fixing them. Until someone makes noise, this is just going to keep happening.
- octopoc 5y ago
  It's a big red flag when a dev makes something really cool that is (according to them) 90% done, then hands it off to someone else to finish. It's a sign that they: 1. Can't finish what they started 2. Are going to rewrite stuff unnecessarily 3. Are willing to throw their coworkers under the bus
  - Aqueous 5y ago
    I will say, in this case it is not what happened. The previous system launched successfully with an incorrect schema. The platform grew and when it came time to add more features, some of them were either impossible to do because of the incorrect schema, or they would have been so complex to implement that the system would have fallen apart. Needless to say, the schema needed to be corrected, but the wrong schema was already baked in all across the platform. So it's not like the main developer handed something off that was unfinished. They handed something off that was finished but wrong. And because of that we had to replace the engine while in flight, so to speak.
- shepherdjerred 5y ago
  I've had this exact same experience. Half-baked, unmaintainable features are launched. The developers get promoted and leave the team. The team is left to clean up and pay the consequences until development speed slows to a crawl. The team isn't able to get promoted because they're busy doing boring, undervalued work. The engineers responsible for the mess ultimately end up ahead. If your goal is to get promoted then clearly the best path is to take shortcuts and not act in the long-term interest of the team. You'll get promoted for it. If your goal is to build a product that won't be burdened in operations/maintenance then you have to move slowly, correctly, and you'll be setting yourself back. This was my experience at a FANG.
  - Aqueous 5y ago
    Precisely. If I were to have neglected to point out the issue, or refused to lead the fix, I would probably have been promoted by now. Instead I chose the hard path and got punished for it. However, the team will benefit far into the future because they will no longer be fighting a schema that was just wrong. The only cost they had to pay was my career velocity.
- gajjanag 5y ago
  And companies wonder why loyalty is hard to find... Far easier to jump ship at opportune moments for fast tracking a career than fighting dysfunctional internal incentive structures.

papito 5y ago

"Complexity kills" is an old mantra a lot of us fossils are familiar with, but the new kids still need to learn this lesson the hard way. What makes things worse is the ability to stitch together twenty different AWS services together because - well - you can. Solving hard problems in simple and maintainable ways, I would argue, IS the job. But what I see is engineers solving problems they don't even have, for THIS IS THE WAY. Also, there is a FAANG guy on the team that claims that's how they did it over there.

scruple 5y ago
The cargo cult mentality is the thing that I've seen throughout my career. FAANG does it this way therefore we must do it this way (I've also heard it parroted at the senior leadership and executive levels more than once!). IMO, this is the driving force behind juniors and intermediates who pull too much complexity into their solutions.
- deckard1 5y ago
  Yes, and I've commented about this again and again. The push for microservices, K8S, Docker. It's all the same thing. It all comes from Google. Or occasionally Facebook. We can thank them for many of the bad ideas of the past 20 years. Open floorplan offices. K8S could only have come from Google. It's a reflection of their corporate structure. Conway's Law and all that. Much like microservices. It's a reflection of their internal chaos. And yet every company that isn't Google tries to emulate them and fails. Then they wonder why microservices suck so much. It's not that microservices suck. It's that Google sucks. Google sucks so much that microservices are pretty much the only way they can operate as an entity.
  - scruple 5y ago
    I was going to type out something about the lack of cohesive technical vision being a massive part of the problem but Conway's Law is far too succinct.
  - deleted 5y ago
    [deleted]

jtdev 5y ago

Technical employees who frequently act like firefighters (especially those that draw attention to themselves when doing so, i.e., the martyr types) are toxic and should be removed ASAP.

rixed 5y ago
No more firefighters, no more fires! Qed.

flatline 5y ago

Is k8s really considered that complex? Yes, you need a lot of tooling to go from code to orchestration. But when you are commoditizing software to run on a desktop, on prem, in the cloud, across many customers and topologies, reproducible deployments are way simpler than hand-rolled whatevers.

logshipper 5y ago
From what I understand, the labour/complexity in K8S lies not with the deployment, but with managing said deployment over the long term. This is a real problem for system architects/security professionals that need to maintain a bird's eye view of the system - a job that gets increasingly complex with every additional microservice once you hit critical mass. This is not to say that K8S does not work - they do, but for certain problems. The cargo-cult mentality, however, threatens to introduce complexity in systems where there is need for none.
paulddraper 5y ago
Not exactly. But k8s enables a lot of complexity that is usually underestimated.
henning 5y ago
Kubernetes itself is not the problem. It's the DIY PaaS setups people build with YAML files, bash scripts, Jenkins jobs and webhooks that people build on top of it.

paulddraper 5y ago

I swear 50% of microservices is feeling really productive as you have invented so many more things that need gluing together.

ahallock 5y ago

Failing upward and job security from complex systems is a strange thing. It's never done outright. Like, I've never had a conversation with another engineer where we conspire or they tell me they're going to or have built a complex system just for this purpose. It's only observed amusingly after the fact.

beebmam 5y ago

People need to stop trying to get promoted then. I'm very happy building simple and easy to maintain systems without flashy features. I couldn't care less about a promotion.