9 ms·

Let It Fail

233 points by bo0tzz 4y ago

abigailserdana 4y ago
I agree with this totally.
duxup 4y ago
One of the first lessons I used to teach new guys in tech support was to work one ticket at a time. Take your time, document….. and just let the phone ring and let SLAs get missed if that is what happens. Go getter guys would rush because calls we’re waiting, it was hard to break them of the habit of trying to rush and documenting later. but inevitably rushing meant poor work was done. In the end though you have to do your job right fist. That’s always on you. Missed calls, ultimately that’s someone else’s staffing choice. Sure someone will say “Hey what happened you missed a call?!?” but if there were 4 people and 6 calls… there you go. In many orgs if the call(s) wasn’t missed it probably would never get addressed…
- toomuchtodo 4y ago
  This. You are only human. The business has a choice; drop work from the queue or increase the worker count (or scale back expectations for work that cannot be parallelized; obligatory Mythical Man Month reference). If you enable a system to operate successfully to great detriment of its components, it’s still operating and no action will be taken. Failure is a resourcing or scheduling problem, not an individual failing. Failure is a symptom of system defects(s) requiring system improvement. Great post. Stay healthy.
  - senectus1 4y ago
    yup, philosophy for life: Failure is acceptable and if you learn from it even encouraged There is no point stressing about things you have no control over You can only do what you can only do, if you can't do it then find someone that can.
    - andirk 4y ago
      "This too shall pass" we would tell each other. You're not getting paid to stress and age like [some animal that ages quickly]. My favorite battle cry of "Fail fast" was from this Microsoft head of something engineering behind Windows Vista. And how.
      - rzzzt 4y ago
        Mayflies can survive in their immature form for years but only live for 1-2 days as adults: https://en.wikipedia.org/wiki/Mayfly https://en.wikipedia.org/wiki/Mayfly
- Negitivefrags 4y ago
  As a manager, I actually find people doing "big picture" thinking quite annoying to deal with. What I want you to do is concentrate on the tasks in front of you and do those well. You shouldn't be caring about the total amount of tasks to do, or if we are going to finish the project on time. That is my problem, not yours. At some point I realised that it's better to have all the issues in the tracker assigned to project managers and never have more than half a dozen on someone at a time because some people just can't seem to handle it. People with a lot of work assigned to them will report how stressed they are and how they are working too hard. When I ask them if they are working more than 40 hours a week they tell me "no". (We don't allow people to work unapproved overtime) And then if you unassign all the work off them and just parcel out the issues they work the same amount of time and produce about the same amount of work and yet are not "overworked" to hear them tell it.
  - hn_version_0023 4y ago
    > As a manager, I actually find people doing "big picture" thinking quite annoying to deal with People aren't cogs in a machine. Your attitude is dehumanizing. I'm in a position where I also want people to concentrate on their tasks, but way more often than not I have something pointed out by a direct report that I missed -- and because I stow my ego at the door, I accept and appreciate that help, and make sure they're commended for it by the larger organization. They wouldn't be able to do that if they didn't see the big picture. A peer of mine has a great quote from a movie whose name I can't recall, that goes something like this: "Sometimes, you treat people like mushrooms. You keep 'em in the dark and feed 'em shit." Seems apropos. Edit: spelling
  - querulous 4y ago
    you know what is more stressful than having too much work? not understanding the work you do have
  - 5tefan 4y ago
    I am often surprised how the big picture manifests itself in nitty gritty details. I want all teams to know the big picture and you must trust their expertise and experience. This is tough to learn to let go but that's why you hired experts in the first place.
swatcoder 4y ago
The missed lesson here is that that company seemed to be a mismanaged nightmare. Lots of orgs are, and they can be a fine place to cut your teeth and learn some things and build a positive reputation with colleagues, but ideally you learn to recognize when a shitshow is a shitshow so you don’t normalize it. Hopefully Max learned that failing systems are sometimes better than flailing teams, but that failing systems shouldn’t be the norm in the first place. Other people (founders? execs? managers?) had done very poor work there and what he did was find a productive way to deal with it.
- taneq 4y ago
  For most of the article I assumed that the “it” that they were letting fail was the project itself, in order to force organisational change, not the incumbent software system.
  - infogrind 4y ago
    +1, the writing could certainly be improved there.
  - andirk 4y ago
    What editor are you using that uses angled quotes on HN plain text comments??
    - taneq 4y ago
      I think I posted that comment from iOS Safari, they iOS keypad does that.
    - quickthrower2 4y ago
      🄸🄼🄿🄾🅂🅂🄸🄱🄻🄴
foobarbecue 4y ago
I sure hope this isn't something in healthcare or transportation or aerospace or ... any industry that matters at all ...
- Brian_K_White 4y ago
  What does that have to do with anything? I don't want swiss cheese code written by an under-resourced operation in my airplane. If anything, the more critical the application, the more this message applies.
- hinkley 4y ago
  You mean like airplanes where they expect them to fail and so every vehicle has a nearly indestructible telemetry box with a homing beacon? Failure is not the problem. Failing the same way and doing nothing about it is the problem.
  - groby_b 4y ago
    Failing in predictable ways is always a problem. Even if it's a novel way of failing. Those boxes are there as last resort, not as an engineering strategy.
    - giaour 4y ago
      Having a last resort is an engineering strategy (and not a very popular one outside of safety critical industries). I've seen plenty of projects that choose not to design or think about handling "unknown unknown" failure vectors because YAGNI
- duxup 4y ago
  I suspect if people were going to or actually died the author would have mentioned it.
- 000ooo000 4y ago
  This is classic HN pedantry. You really think these people would allow people to die to prove the point in the article? That they couldn't exercise sensible discretion on when failure might be the best course of action?
bruce511 4y ago
Replacing one system (typically referred to as legacy - almost in a pejorative way) with another new system can be a big improvement. Or it can be a complete waste of time and money. It all depends on the actual quality of the old system, and the actual quality of the new system. The most common way this manifests is with technology changes. We must rewrite in a modern language like c#. We must switch our JavaScript framework to React. From my experience 90% of these transitions fail precisely because the legacy system is working fine, and _doesn't_ fail. The new system is always 2 years behind, so is never adopted. Good architecture trumps new language. And there are a lot of legacy systems with good architecture. There are also plenty of systems with bad architecture. Ones with too much tech debt. Ones that will fail. And these need to be rebuilt, hopefully with better architecture. So how does a non-technical person tell the difference (hint: they can't). So they rely on their tech team to tell them. But tech teams want to play with new toys all the time so they're not easily trusted. Changing has huge consequences (usability, training, deployment etc) all of which the tech guys don't care about. The sweet spot is a well architected legacy system, managed by people who both understand tech, and understand the needs of the business,and importantly, understand the architecture so can limit the tech debt created. Such systems exist, but it's rare. And ironically all too often management doesn't even know they have it, so some upstart convinces them to "rewrite in React".
- Szpadel 4y ago
  Agree, but there is also one more point, always allow people that created legacy system to work on new one. In one of projects I worked there was legacy system that had enough issues causes by architecture, that 95% of workforce was assigned into fixing new and new bugs/regressions. So business decided to rewrite that system, but mistake they made was to create complete new team for that task. The new team didn't know why previous solution failed, as this was on first sight very simple system, and not knowing all of the hidden complexity made similar mistakes that caused previous system to fail. And in that way we ended up brand new legacy system that also failed in similar ways. Fortunately I convinced them to create third version with the same team as this second failed system and it all looked well until.. some well known consulting company was hired and they decided to rewrite everything on react, and 90% of people just resigned from work.
larsnystrom 4y ago
To me, this article was less about letting something fail and more about letting people learn for themselves. And I agree: It’s very hard to explain a solution to people when they haven’t fully understood the pain points you’re trying to solve and/or avoid.
- ilaksh 4y ago
  Its just a lack of respect on the part of management for engineering. The engineers clearly stated that the new system was necessary. The managers just refused to respect their expertise or the integrity of the technology.
jasonjmcghee 4y ago
Highly recommend reading The Phoenix Project. Much longer winded than the post, but drives home similar, and a number of other, valuable lessons along the way in a memorable story-setting.
collaborative 4y ago
>Consider that if our hero disappears on a months-long backpacking adventure in Europe and without her to save the day our velocity slows to a crawl, we've stumbled upon a systemic reliance that reveals deeper fragility. >As such generally we want to discourage a culture of heroism and seek to build systems and processes that don't require it. The economy runs on heroism. If you don't believe me, see how central planning went in the last century I have seen heroes quit because of decreased comp leading to huge revenue loss. I have also seen extremely expensive but predictable (often in that they don't deliver) "systems and processes" This post speaks to cushy managerialism
- WJW 4y ago
  What's the alternative to process and knowledge sharing then? If you allow heroism to run rampant in your company you basically guarantee it will die just because a single employee quits, because all areas of the company are rife with single points of failure.
  - ramphastidae 4y ago
    Pay the heroes what they’re worth. Imagine if the Bulls let Michael Jordan join another team because they wanted to discourage heroism.
    - WJW 4y ago
      No amount of pay can stop people from getting hit by a bus or leaving for other reasons though. Michael Jordan is an interesting example because he eventually did stop playing for the Bulls (because he retired) and then it took the Bulls six years before they made the playoffs again. More in general: everyone thinks they're the "hero" in the situation and that they're underappreciated, but for the company the team is (and should be) more important than the individual. Imagine if you got laid off because the "hero" in (say) the sales team died and now the company is no longer profitable, because he/she was the only one bringing in any orders. Would you consider that a well-run company?
      - 4y ago
the_gipsy 4y ago
I disagree. I have worked through exactly this proposition and I just left. Why would anyone that respects themselves work through a project that is set up to fail? If upper management cannot make up their internal politics, I don't want to be someone's pawn to be sacrificed for the greater good. After all, the right decision could be made from the beginning.
taskqa 4y ago
[dead]
xyzzy4747 4y ago
The biggest point of failure actually has nothing to do with the software itself - it’s whether enough money is coming in the door or not. The most backwards and tech debt riddled system will survive if there is enough customer demand and relationships. Meanwhile a perfectly working software project will die or go dormant if the maintainers aren’t getting paid.
JOnAgain 4y ago
I 100% agree. I've seen this happen to many managers at FAANG. You're promised 10 people for a project next year, you get 4. So you cut, manage, optimize, engineer, and hustle to get something done and working. Management above you sees this as relative success, and validates that they were right not to give you more people. Disillusionment sets in, the team quits or transfers, you get bad reviews. The next year, you're fired or transferred, and the new guy comes in, he gets 10 people, saves the day and gets high praise. He gets good reviews and maybe a promotion. You look back with resentment. The problem wouldn't have been there if they had given you the resources you said you need. If you had 10 people, you also could have saved the day. This is the sickness of large organizations and non-technical management. I have rarely seen organizations not run by owners spend the $1 on prevention today, they almost always elect for the $10 for a fix later. If you need 10 people to do a job, and you get 4, the job should fail. Your systems should be going down. Don't sabotage, but upper management needs to see, not hear about, failure. Update deliverable timelines form 6 months to 18 months or just cancel new projects all together, change your SLA's from 99.9% to 90%. Change your ticket response times from 30 minutes to 12 hours or best effort business hours. You'll either get the resources, get attention for reprioritization, or learn that your job isn't a real priority.
- abigailserdana 4y ago
  Something similar happened in my previous organisation with less than 5 people on the engineering side where I was asked to do a major refactor of the codebase that touched 70% of the files while also adding new features(with changing requirements) at the same time. Though I was working day and night, it took me thrice the time to do it than we initially expected. I was put on PIP after delivering the work(prod deployed) so I left. Then they got 3 engineers from top 1 percentile of industry to work on the same codebase. All this time I was shouting that I need someone to help me in the team but got no one :) . Question mark was put on my performance rather than the management of the team/resources.
- bookish 4y ago
  The right choice between the "get something done and working" and "your systems should be going down" paths might depend on the situation. Sometimes the right call for the company is to actually have "relative success" despite the downsides if it's better than the downsides of the early failure. Implicitly, this article implies some communication failures in an organization, where visible failures are used as as attention-getting substitute for what should be healthy communication and prioritization channels.
dsugarman 4y ago
Major refactoring of a platform is almost never the right answer, wholesale replacing battle tested code with theoretically better code satisfies only engineering teams and kills businesses.
- swagasaurus-rex 4y ago
  Sure just let your competitors do that hard work instead. A critical production piece of code can be replaced safely if a comprehensive test suite can ensure feature parity. It’s a lot of work but if people rely on it may as well test anyways. The tests will survive the upgrade.

233 points by bo0tzz 4y ago

I agree with this totally.

One of the first lessons I used to teach new guys in tech support was to work one ticket at a time. Take your time, document….. and just let the phone ring and let SLAs get missed if that is what happens. Go getter guys would rush because calls we’re waiting, it was hard to break them of the habit of trying to rush and documenting later. but inevitably rushing meant poor work was done. In the end though you have to do your job right fist. That’s always on you. Missed calls, ultimately that’s someone else’s staffing choice. Sure someone will say “Hey what happened you missed a call?!?” but if there were 4 people and 6 calls… there you go. In many orgs if the call(s) wasn’t missed it probably would never get addressed…

toomuchtodo 4y ago
This. You are only human. The business has a choice; drop work from the queue or increase the worker count (or scale back expectations for work that cannot be parallelized; obligatory Mythical Man Month reference). If you enable a system to operate successfully to great detriment of its components, it’s still operating and no action will be taken. Failure is a resourcing or scheduling problem, not an individual failing. Failure is a symptom of system defects(s) requiring system improvement. Great post. Stay healthy.
- senectus1 4y ago
  yup, philosophy for life: Failure is acceptable and if you learn from it even encouraged There is no point stressing about things you have no control over You can only do what you can only do, if you can't do it then find someone that can.
  - andirk 4y ago
    "This too shall pass" we would tell each other. You're not getting paid to stress and age like [some animal that ages quickly]. My favorite battle cry of "Fail fast" was from this Microsoft head of something engineering behind Windows Vista. And how.
    - rzzzt 4y ago
      Mayflies can survive in their immature form for years but only live for 1-2 days as adults: https://en.wikipedia.org/wiki/Mayfly https://en.wikipedia.org/wiki/Mayfly
Negitivefrags 4y ago
As a manager, I actually find people doing "big picture" thinking quite annoying to deal with. What I want you to do is concentrate on the tasks in front of you and do those well. You shouldn't be caring about the total amount of tasks to do, or if we are going to finish the project on time. That is my problem, not yours. At some point I realised that it's better to have all the issues in the tracker assigned to project managers and never have more than half a dozen on someone at a time because some people just can't seem to handle it. People with a lot of work assigned to them will report how stressed they are and how they are working too hard. When I ask them if they are working more than 40 hours a week they tell me "no". (We don't allow people to work unapproved overtime) And then if you unassign all the work off them and just parcel out the issues they work the same amount of time and produce about the same amount of work and yet are not "overworked" to hear them tell it.
- hn_version_0023 4y ago
  > As a manager, I actually find people doing "big picture" thinking quite annoying to deal with People aren't cogs in a machine. Your attitude is dehumanizing. I'm in a position where I also want people to concentrate on their tasks, but way more often than not I have something pointed out by a direct report that I missed -- and because I stow my ego at the door, I accept and appreciate that help, and make sure they're commended for it by the larger organization. They wouldn't be able to do that if they didn't see the big picture. A peer of mine has a great quote from a movie whose name I can't recall, that goes something like this: "Sometimes, you treat people like mushrooms. You keep 'em in the dark and feed 'em shit." Seems apropos. Edit: spelling
- querulous 4y ago
  you know what is more stressful than having too much work? not understanding the work you do have
- 5tefan 4y ago
  I am often surprised how the big picture manifests itself in nitty gritty details. I want all teams to know the big picture and you must trust their expertise and experience. This is tough to learn to let go but that's why you hired experts in the first place.

swatcoder 4y ago

The missed lesson here is that that company seemed to be a mismanaged nightmare. Lots of orgs are, and they can be a fine place to cut your teeth and learn some things and build a positive reputation with colleagues, but ideally you learn to recognize when a shitshow is a shitshow so you don’t normalize it. Hopefully Max learned that failing systems are sometimes better than flailing teams, but that failing systems shouldn’t be the norm in the first place. Other people (founders? execs? managers?) had done very poor work there and what he did was find a productive way to deal with it.

taneq 4y ago
For most of the article I assumed that the “it” that they were letting fail was the project itself, in order to force organisational change, not the incumbent software system.
- infogrind 4y ago
  +1, the writing could certainly be improved there.
- andirk 4y ago
  What editor are you using that uses angled quotes on HN plain text comments??
  - taneq 4y ago
    I think I posted that comment from iOS Safari, they iOS keypad does that.
  - quickthrower2 4y ago
    🄸🄼🄿🄾🅂🅂🄸🄱🄻🄴

foobarbecue 4y ago

I sure hope this isn't something in healthcare or transportation or aerospace or ... any industry that matters at all ...

Brian_K_White 4y ago
What does that have to do with anything? I don't want swiss cheese code written by an under-resourced operation in my airplane. If anything, the more critical the application, the more this message applies.
hinkley 4y ago
You mean like airplanes where they expect them to fail and so every vehicle has a nearly indestructible telemetry box with a homing beacon? Failure is not the problem. Failing the same way and doing nothing about it is the problem.
- groby_b 4y ago
  Failing in predictable ways is always a problem. Even if it's a novel way of failing. Those boxes are there as last resort, not as an engineering strategy.
  - giaour 4y ago
    Having a last resort is an engineering strategy (and not a very popular one outside of safety critical industries). I've seen plenty of projects that choose not to design or think about handling "unknown unknown" failure vectors because YAGNI
duxup 4y ago
I suspect if people were going to or actually died the author would have mentioned it.
000ooo000 4y ago
This is classic HN pedantry. You really think these people would allow people to die to prove the point in the article? That they couldn't exercise sensible discretion on when failure might be the best course of action?

bruce511 4y ago

Replacing one system (typically referred to as legacy - almost in a pejorative way) with another new system can be a big improvement. Or it can be a complete waste of time and money. It all depends on the actual quality of the old system, and the actual quality of the new system. The most common way this manifests is with technology changes. We must rewrite in a modern language like c#. We must switch our JavaScript framework to React. From my experience 90% of these transitions fail precisely because the legacy system is working fine, and _doesn't_ fail. The new system is always 2 years behind, so is never adopted. Good architecture trumps new language. And there are a lot of legacy systems with good architecture. There are also plenty of systems with bad architecture. Ones with too much tech debt. Ones that will fail. And these need to be rebuilt, hopefully with better architecture. So how does a non-technical person tell the difference (hint: they can't). So they rely on their tech team to tell them. But tech teams want to play with new toys all the time so they're not easily trusted. Changing has huge consequences (usability, training, deployment etc) all of which the tech guys don't care about. The sweet spot is a well architected legacy system, managed by people who both understand tech, and understand the needs of the business,and importantly, understand the architecture so can limit the tech debt created. Such systems exist, but it's rare. And ironically all too often management doesn't even know they have it, so some upstart convinces them to "rewrite in React".

Szpadel 4y ago
Agree, but there is also one more point, always allow people that created legacy system to work on new one. In one of projects I worked there was legacy system that had enough issues causes by architecture, that 95% of workforce was assigned into fixing new and new bugs/regressions. So business decided to rewrite that system, but mistake they made was to create complete new team for that task. The new team didn't know why previous solution failed, as this was on first sight very simple system, and not knowing all of the hidden complexity made similar mistakes that caused previous system to fail. And in that way we ended up brand new legacy system that also failed in similar ways. Fortunately I convinced them to create third version with the same team as this second failed system and it all looked well until.. some well known consulting company was hired and they decided to rewrite everything on react, and 90% of people just resigned from work.

larsnystrom 4y ago

To me, this article was less about letting something fail and more about letting people learn for themselves. And I agree: It’s very hard to explain a solution to people when they haven’t fully understood the pain points you’re trying to solve and/or avoid.

ilaksh 4y ago
Its just a lack of respect on the part of management for engineering. The engineers clearly stated that the new system was necessary. The managers just refused to respect their expertise or the integrity of the technology.

jasonjmcghee 4y ago

Highly recommend reading The Phoenix Project. Much longer winded than the post, but drives home similar, and a number of other, valuable lessons along the way in a memorable story-setting.

collaborative 4y ago

>Consider that if our hero disappears on a months-long backpacking adventure in Europe and without her to save the day our velocity slows to a crawl, we've stumbled upon a systemic reliance that reveals deeper fragility. >As such generally we want to discourage a culture of heroism and seek to build systems and processes that don't require it. The economy runs on heroism. If you don't believe me, see how central planning went in the last century I have seen heroes quit because of decreased comp leading to huge revenue loss. I have also seen extremely expensive but predictable (often in that they don't deliver) "systems and processes" This post speaks to cushy managerialism

WJW 4y ago
What's the alternative to process and knowledge sharing then? If you allow heroism to run rampant in your company you basically guarantee it will die just because a single employee quits, because all areas of the company are rife with single points of failure.
- ramphastidae 4y ago
  Pay the heroes what they’re worth. Imagine if the Bulls let Michael Jordan join another team because they wanted to discourage heroism.
  - WJW 4y ago
    No amount of pay can stop people from getting hit by a bus or leaving for other reasons though. Michael Jordan is an interesting example because he eventually did stop playing for the Bulls (because he retired) and then it took the Bulls six years before they made the playoffs again. More in general: everyone thinks they're the "hero" in the situation and that they're underappreciated, but for the company the team is (and should be) more important than the individual. Imagine if you got laid off because the "hero" in (say) the sales team died and now the company is no longer profitable, because he/she was the only one bringing in any orders. Would you consider that a well-run company?
    - 4y ago

the_gipsy 4y ago

I disagree. I have worked through exactly this proposition and I just left. Why would anyone that respects themselves work through a project that is set up to fail? If upper management cannot make up their internal politics, I don't want to be someone's pawn to be sacrificed for the greater good. After all, the right decision could be made from the beginning.

taskqa 4y ago

[dead]

xyzzy4747 4y ago

The biggest point of failure actually has nothing to do with the software itself - it’s whether enough money is coming in the door or not. The most backwards and tech debt riddled system will survive if there is enough customer demand and relationships. Meanwhile a perfectly working software project will die or go dormant if the maintainers aren’t getting paid.

JOnAgain 4y ago

I 100% agree. I've seen this happen to many managers at FAANG. You're promised 10 people for a project next year, you get 4. So you cut, manage, optimize, engineer, and hustle to get something done and working. Management above you sees this as relative success, and validates that they were right not to give you more people. Disillusionment sets in, the team quits or transfers, you get bad reviews. The next year, you're fired or transferred, and the new guy comes in, he gets 10 people, saves the day and gets high praise. He gets good reviews and maybe a promotion. You look back with resentment. The problem wouldn't have been there if they had given you the resources you said you need. If you had 10 people, you also could have saved the day. This is the sickness of large organizations and non-technical management. I have rarely seen organizations not run by owners spend the $1 on prevention today, they almost always elect for the $10 for a fix later. If you need 10 people to do a job, and you get 4, the job should fail. Your systems should be going down. Don't sabotage, but upper management needs to see, not hear about, failure. Update deliverable timelines form 6 months to 18 months or just cancel new projects all together, change your SLA's from 99.9% to 90%. Change your ticket response times from 30 minutes to 12 hours or best effort business hours. You'll either get the resources, get attention for reprioritization, or learn that your job isn't a real priority.

abigailserdana 4y ago
Something similar happened in my previous organisation with less than 5 people on the engineering side where I was asked to do a major refactor of the codebase that touched 70% of the files while also adding new features(with changing requirements) at the same time. Though I was working day and night, it took me thrice the time to do it than we initially expected. I was put on PIP after delivering the work(prod deployed) so I left. Then they got 3 engineers from top 1 percentile of industry to work on the same codebase. All this time I was shouting that I need someone to help me in the team but got no one :) . Question mark was put on my performance rather than the management of the team/resources.
bookish 4y ago
The right choice between the "get something done and working" and "your systems should be going down" paths might depend on the situation. Sometimes the right call for the company is to actually have "relative success" despite the downsides if it's better than the downsides of the early failure. Implicitly, this article implies some communication failures in an organization, where visible failures are used as as attention-getting substitute for what should be healthy communication and prioritization channels.

dsugarman 4y ago

Major refactoring of a platform is almost never the right answer, wholesale replacing battle tested code with theoretically better code satisfies only engineering teams and kills businesses.

swagasaurus-rex 4y ago
Sure just let your competitors do that hard work instead. A critical production piece of code can be replaced safely if a comprehensive test suite can ensure feature parity. It’s a lot of work but if people rely on it may as well test anyways. The tests will survive the upgrade.