9 ms·

Fixing all the bugs won't solve all the problems – Deming's path of frustration

deleted 3y ago
[deleted]
skmurphy 3y ago
When Frank Marshall was hired as VP of engineering at Cisco in 1992, he assessed after three months that we had a software quality problem and forbade any software engineering activity except "Fixing bugs." This went on for perhaps six months. It allowed key competitors to get ahead of us because we could not address some serious architectural issues in the code base because that would be writing new code, not fixing bugs. Gerald Weinberg talks about "Software Stalins," managers who grab onto one indicator and think that driving it to zero (or 100, or 11 for you Spinal Tap Fans) will resolve all other problems. If only things were so simple.
- throwiforgtnlzy 3y ago
  The lack of nuance to consider exceptions appears to have been the problem. Perhaps a lot of products needed bug fix campaigns because most corporations tend to reward SWE productivity based on so-called "evidence" like: - KLoC - # of commits - adding features - so-called "impact" , rather than: - reducing the cost to maintain code - making code more robust - making features actually work as users expect - rip-replacing what shouldn't be salvaged because it's too expensive So instead of adopting a funny mustache and making inflexible commandments, it would be less disruptive to have targeted bug fix campaigns where needed to boost nonfunctional requirements like quality, reducing support costs, and improving user satisfaction to their desired target levels.
- swatcoder 3y ago
  This is a really interesting anecdote because (obviously) Cisco fared quite well on net over the last severl decades, is largely respected as a reliable vendor for critical infrastructure, and doesn't evidence even a hiccup in market assessments of its health during that time. https://www.macrotrends.net/stocks/charts/CSCO/cisco/stock-price-history https://www.macrotrends.net/stocks/charts/CSCO/cisco/stock-p... I don't mean to take away the truth of your experience or dismiss that you might better know the disconnect between what you describe and what outsiders might see in hindsight -- it's just the disconnect itself that's fascinating. > If only things were so simple. Indeed!
  - skmurphy 3y ago
    The renewed growth and increasing customer satisfaction--and our long-term success--were due to engineering abandoning a poorly thought-out policy before it crippled us instead of merely hampering us. Part of the challenge in a "freeze spec" approach implicit in "only fix bugs" is that it assumes the market requirements are well established. That was definitely not the case for Internet protocols/functionality in 1992. HTTP traffic did not exist yet (along with a couple of dozen other things that would quickly become dominant by 1997. The effect of the bug fix stall was to give our more nimble competitors more growth and more runway. The fact that Cisco did very well was despite the "just fix bugs" mandate. I offered the story because I thought some folks reading the linked post would find it unbelievable that a company would decide to fixate on a single simple metric when operating in a complex and evolving environment.
    - kazinator 3y ago
      You have to do both, on different branches. Rearchitect things in one branch, fix nothing but bugs in another. Forward port all the bugfixes you can from one to the other. Though: version control ... 1992 ... haha. Hey, that's when that unwieldy behemoth ClearCase was released.
      - pjc50 3y ago
        I think that approach would be doomed even in the present day. Multiple "legacy" release branches is very hard to manage. Only the very largest such as Microsoft can do it, and they clearly hate doing so and want to push to rolling releases like everyone else.
        worthless-trash 3y ago
        Red Hat / RHEL does this on all EUS release every day, on RHEL 6,7,8,9. Tell me more about how you would subscribe to a MS "bugfix only channel of updates" ? This is my first time hearing about that.
- shiroiushi 3y ago
  It's too bad that Microsoft didn't take this same approach, and stick with it for a long time. I'm wondering what things are like in the alternate universe where OS/2 became the dominant PC OS instead of Windows.
- sgift 3y ago
  Architectural problems are bugs. Hopefully, most companies have VP of engineering who understand that.
  - Attrecomet 3y ago
    Disagree. Architecture problems are different enough from software bugs that it's worth it to keep them separate words.
    - chuckadams 3y ago
      If bad architecture makes the system fall over, that's a bug. It's not black and white.
- IAmYourDensity 3y ago
  > "Software Stalins," managers who grab onto one indicator and think that driving it to zero (or 100, or 11 for you Spinal Tap Fans) will resolve all other problems. Paul O’Neill famously did exactly this at Alcoa starting in 1987, focusing solely on worker safety resolved many other problems and multiplied profitability. >The company's market value increased from $3 billion in 1986 to $27.53 billion in 2000, while net income increased from $200 million to $1.484 billion. [0] https://www.forbes.com/sites/roddwagner/2019/01/22/have-we-learned-the-alcoa-keystone-habit-lesson/ https://www.forbes.com/sites/roddwagner/2019/01/22/have-we-l...
  - skmurphy 3y ago
    It's a reasonable counter-example, two other CEOs of successful firms who emphasized safety were Lawrence Culp at Danaher (he is now at GE) and David Cote at Honeywell. All three had tenures more than a decade where their firms invested in and introduced new products, so I suspect they focused on more than worker safety but I am not personally familiar with their policies and decisions. I think it makes a very good story that a focus on worker safety--to the exclusion of any other objectives--is all that you need. But I don't think anything is that simple.
jeffrey-sherman 3y ago
I recently witnessed an argument between a dev manager and the devops team lead. The manager manager was convinced that fixing all of the bugs would lead to perfect, 100% successful execution. The devops team lead spent an hour trying to explain why the software couldn’t achieve better reliability than AWS (the system spread across AWS zones, but each instance was contained within a single zone).
- yen223 3y ago
  100% successful execution is possible, assuming 1. You can guarantee no bugs in your system 2. You can guarantee no bugs in AWS's systems 3. You can guarantee there's no nuclear attacks on us-east-1 within the next year or so
  - boxed 3y ago
    No cosmic rays ;)
    - throwaway290 3y ago
      And no network cables or ISPs...
  - pjc50 3y ago
    You've entirely forgotten routine hardware failure. At large enough scale all sorts of weird stuff happens.
- klowner 3y ago
  This sounds like a possible explanation for why I keep getting recommended different AWS client libraries from the boss to "solve instability" of a service that works 99.8% except for when the S3 endpoint (backblaze) has brief mystery issues.
musicale 3y ago
Design (and specification) flaws are harder to fix than regular bugs.
datascienced 3y ago
Well yes. Define bugs to be application issues only and not devops issues and you can only go so far by tackling the app bugs. The end user expedience depends on all elements of the system: including app, devops and human interactions!
- sloowm 3y ago
  Yeah, the article also transitions from quality to bugs without explaining why quality is the lack of bugs instead of within bounds loading times, a good UI/UX, and relevant features. Fixing bugs also seems to be limited to try catching all bugs instead of making changes to have the program do the correct thing like only loading the necessary data.
- jamil7 3y ago
  I had the exact same thought, this was a very weird article.
QuantumG 3y ago
I just like fixing bugs. If you have someone in your team like that, give them latitude.
- dgellow 3y ago
  That’s me, I love fixing software. Just give me a list of bug tickets to go through and I’m happy for some time. I think we deserve a cool title, Senior Bug Hunter would be pretty neat on a resume :)
eimrine 3y ago
> The software runs slowly because large amounts of unnecessary data are being sent to the users /s/to/from
onion2k 3y ago
I think the lesson from the article is that we should be things about issues like slow loading speed, overly chunky page weight, lag etc as bugs. The reason so many teams don't fix these problems is because they frame them as business problems instead of tech problems. We've all seen a team lament "our app would be fast if only we were given time to fix it!" By framing loading time delays as a bug, you're much more likely to get that time.
a_c 3y ago
The biggest bug is the thinking that your proverbial calendar app is the next billion dollar application but in reality no one gives a wink. That's the root of all problems
- skmurphy 3y ago
  Cisco ultimately met with a fair amount of success in the market. Their Internet Operating System was certainly a 10B application, perhaps worth 100B.
nanolith 3y ago
Dijkstra grappled with these issues. Two classic notes of his are: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/EWD303.html https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/E... https://www.cs.utexas.edu/~EWD/transcriptions/EWD10xx/EWD1036.html https://www.cs.utexas.edu/~EWD/transcriptions/EWD10xx/EWD103... There are several takeaways here, but three of my favorite highlights: * Dijkstra actually did not like calling such errors "bugs", as it is a framing problem. * Likewise, he believed that we should avoid anthropomorphizing software or identifying with it. * Finally, he believed in building a formal specification of software and then proving that the software matched this specification. Part of this was to avoid "bug fixing" hunts in which the focus was on "debugging" instead of correctness. But also, this was to ensure that there was a proper system view of the software, which ties back into the conclusion of the article here.
- rramadass 3y ago
  Nice! It is a tragedy that programmers nowadays don't read Dijkstra nor think about what he wrote and meant. I tell people to think of him as their "Cus D'Amato" if they want to be a "Mike Tyson" in their field i.e. a man who has thought deeply about the subject, knows all the angles and can "train" one in the "correct" manner of writing programs.
- shiandow 3y ago
  Interestingly most bugs I encounter are bugs in the specification, so in some ways formal specification just moves the problem.
  - nanolith 3y ago
    Formal specification means building up this specification from theory, and building proofs along the way. Of course, there can always be errors in specification, but this is supposed to be the first place we implement SAT solvers or proof assistants. During Dijkstra's time, such technologies were not yet available as they are now. There is work to be done until this is all practical and the overhead to do this is within the current overhead of software engineering, but we are quickly reaching that point in time.
eviks 3y ago
Don't understand why this can't be called a bug > The software runs slowly because large amounts of unnecessary data are being sent to the users.
- regularfry 3y ago
  Depends if it's a consequence of intentional design or not. It's useful to have a separate designation for accidents of implementation.
  - boxed 3y ago
    "Unnecessary" was the word in the blog post. That word implies it can be just straight up fixed.
    - regularfry 3y ago
      Not necessarily. "Unnecessary" to the client doesn't mean that the server can know that ahead of sending it. A redesign can be the only way to get out of that.
ChrisMarshallNY 3y ago
Reminds me of McConnell’s Software Quality at Top Speed essay, from almost 30 years ago[0]. This chart, in particular[1]. These issues are fundamental, and persist, regardless of the technology, or state of the art. plus ça change [0] https://stevemcconnell.com/articles/software-quality-at-top-speed/ https://stevemcconnell.com/articles/software-quality-at-top-... [1] https://stevemcconnell.com/wp-content/uploads/2017/08/art04-1.gif https://stevemcconnell.com/wp-content/uploads/2017/08/art04-...
pfdietz 3y ago
Also, even if you just look at bugs, they will level off at some equilibrium level if the software is being changed (beyond fixing bugs), since change introduces bugs.
hasoleju 3y ago
There is always a tradeoff between different types of use cases. Some users solve problem A with the sofware. An other group solves problem B. Using the same software in both cases leads to edge cases that are bugs for one group but cannot be fixed without introducing bugs for the second group. The clear specification really is the solution to resolve this issue. That you don't declare behavior as buggy that is according to specification.
- jfil 3y ago
  You may hit a point where the software can't balance the needs of problem A and problem B users. Then, the solution is to split the product into 2 distinct ones. Or to exit the market for problem A in order to focus on delivering a solid solution for problem B.
w10-1 3y ago
It's not really bugs vs design issues. It's more changeable vs unchangeable. Part of that is technical cost, but in my experience most is perceived risk, which is largely a function of understanding and team (leader) confidence. The scariest things are existential: telling people the system they believe in has fatal flaws introduces a gap that can only be filled with a lot of confidence, which is typically lacking when growing fast or with new systems. The main remedy is to be sure you have an actual view and understanding of the system, and address things from that perspective. Otherwise, bugs are just make-work.

deleted 3y ago

[deleted]

When Frank Marshall was hired as VP of engineering at Cisco in 1992, he assessed after three months that we had a software quality problem and forbade any software engineering activity except "Fixing bugs." This went on for perhaps six months. It allowed key competitors to get ahead of us because we could not address some serious architectural issues in the code base because that would be writing new code, not fixing bugs. Gerald Weinberg talks about "Software Stalins," managers who grab onto one indicator and think that driving it to zero (or 100, or 11 for you Spinal Tap Fans) will resolve all other problems. If only things were so simple.

throwiforgtnlzy 3y ago
The lack of nuance to consider exceptions appears to have been the problem. Perhaps a lot of products needed bug fix campaigns because most corporations tend to reward SWE productivity based on so-called "evidence" like: - KLoC - # of commits - adding features - so-called "impact" , rather than: - reducing the cost to maintain code - making code more robust - making features actually work as users expect - rip-replacing what shouldn't be salvaged because it's too expensive So instead of adopting a funny mustache and making inflexible commandments, it would be less disruptive to have targeted bug fix campaigns where needed to boost nonfunctional requirements like quality, reducing support costs, and improving user satisfaction to their desired target levels.
swatcoder 3y ago
This is a really interesting anecdote because (obviously) Cisco fared quite well on net over the last severl decades, is largely respected as a reliable vendor for critical infrastructure, and doesn't evidence even a hiccup in market assessments of its health during that time. https://www.macrotrends.net/stocks/charts/CSCO/cisco/stock-price-history https://www.macrotrends.net/stocks/charts/CSCO/cisco/stock-p... I don't mean to take away the truth of your experience or dismiss that you might better know the disconnect between what you describe and what outsiders might see in hindsight -- it's just the disconnect itself that's fascinating. > If only things were so simple. Indeed!
- skmurphy 3y ago
  The renewed growth and increasing customer satisfaction--and our long-term success--were due to engineering abandoning a poorly thought-out policy before it crippled us instead of merely hampering us. Part of the challenge in a "freeze spec" approach implicit in "only fix bugs" is that it assumes the market requirements are well established. That was definitely not the case for Internet protocols/functionality in 1992. HTTP traffic did not exist yet (along with a couple of dozen other things that would quickly become dominant by 1997. The effect of the bug fix stall was to give our more nimble competitors more growth and more runway. The fact that Cisco did very well was despite the "just fix bugs" mandate. I offered the story because I thought some folks reading the linked post would find it unbelievable that a company would decide to fixate on a single simple metric when operating in a complex and evolving environment.
  - kazinator 3y ago
    You have to do both, on different branches. Rearchitect things in one branch, fix nothing but bugs in another. Forward port all the bugfixes you can from one to the other. Though: version control ... 1992 ... haha. Hey, that's when that unwieldy behemoth ClearCase was released.
    - pjc50 3y ago
      I think that approach would be doomed even in the present day. Multiple "legacy" release branches is very hard to manage. Only the very largest such as Microsoft can do it, and they clearly hate doing so and want to push to rolling releases like everyone else.
      - worthless-trash 3y ago
        Red Hat / RHEL does this on all EUS release every day, on RHEL 6,7,8,9. Tell me more about how you would subscribe to a MS "bugfix only channel of updates" ? This is my first time hearing about that.
shiroiushi 3y ago
It's too bad that Microsoft didn't take this same approach, and stick with it for a long time. I'm wondering what things are like in the alternate universe where OS/2 became the dominant PC OS instead of Windows.
sgift 3y ago
Architectural problems are bugs. Hopefully, most companies have VP of engineering who understand that.
- Attrecomet 3y ago
  Disagree. Architecture problems are different enough from software bugs that it's worth it to keep them separate words.
  - chuckadams 3y ago
    If bad architecture makes the system fall over, that's a bug. It's not black and white.
IAmYourDensity 3y ago
> "Software Stalins," managers who grab onto one indicator and think that driving it to zero (or 100, or 11 for you Spinal Tap Fans) will resolve all other problems. Paul O’Neill famously did exactly this at Alcoa starting in 1987, focusing solely on worker safety resolved many other problems and multiplied profitability. >The company's market value increased from $3 billion in 1986 to $27.53 billion in 2000, while net income increased from $200 million to $1.484 billion. [0] https://www.forbes.com/sites/roddwagner/2019/01/22/have-we-learned-the-alcoa-keystone-habit-lesson/ https://www.forbes.com/sites/roddwagner/2019/01/22/have-we-l...
- skmurphy 3y ago
  It's a reasonable counter-example, two other CEOs of successful firms who emphasized safety were Lawrence Culp at Danaher (he is now at GE) and David Cote at Honeywell. All three had tenures more than a decade where their firms invested in and introduced new products, so I suspect they focused on more than worker safety but I am not personally familiar with their policies and decisions. I think it makes a very good story that a focus on worker safety--to the exclusion of any other objectives--is all that you need. But I don't think anything is that simple.

jeffrey-sherman 3y ago

I recently witnessed an argument between a dev manager and the devops team lead. The manager manager was convinced that fixing all of the bugs would lead to perfect, 100% successful execution. The devops team lead spent an hour trying to explain why the software couldn’t achieve better reliability than AWS (the system spread across AWS zones, but each instance was contained within a single zone).

yen223 3y ago
100% successful execution is possible, assuming 1. You can guarantee no bugs in your system 2. You can guarantee no bugs in AWS's systems 3. You can guarantee there's no nuclear attacks on us-east-1 within the next year or so
- boxed 3y ago
  No cosmic rays ;)
  - throwaway290 3y ago
    And no network cables or ISPs...
- pjc50 3y ago
  You've entirely forgotten routine hardware failure. At large enough scale all sorts of weird stuff happens.
klowner 3y ago
This sounds like a possible explanation for why I keep getting recommended different AWS client libraries from the boss to "solve instability" of a service that works 99.8% except for when the S3 endpoint (backblaze) has brief mystery issues.

musicale 3y ago

Design (and specification) flaws are harder to fix than regular bugs.

datascienced 3y ago

Well yes. Define bugs to be application issues only and not devops issues and you can only go so far by tackling the app bugs. The end user expedience depends on all elements of the system: including app, devops and human interactions!

sloowm 3y ago
Yeah, the article also transitions from quality to bugs without explaining why quality is the lack of bugs instead of within bounds loading times, a good UI/UX, and relevant features. Fixing bugs also seems to be limited to try catching all bugs instead of making changes to have the program do the correct thing like only loading the necessary data.
jamil7 3y ago
I had the exact same thought, this was a very weird article.

QuantumG 3y ago

I just like fixing bugs. If you have someone in your team like that, give them latitude.

dgellow 3y ago
That’s me, I love fixing software. Just give me a list of bug tickets to go through and I’m happy for some time. I think we deserve a cool title, Senior Bug Hunter would be pretty neat on a resume :)

eimrine 3y ago

> The software runs slowly because large amounts of unnecessary data are being sent to the users /s/to/from

onion2k 3y ago

I think the lesson from the article is that we should be things about issues like slow loading speed, overly chunky page weight, lag etc as bugs. The reason so many teams don't fix these problems is because they frame them as business problems instead of tech problems. We've all seen a team lament "our app would be fast if only we were given time to fix it!" By framing loading time delays as a bug, you're much more likely to get that time.

a_c 3y ago

The biggest bug is the thinking that your proverbial calendar app is the next billion dollar application but in reality no one gives a wink. That's the root of all problems

skmurphy 3y ago
Cisco ultimately met with a fair amount of success in the market. Their Internet Operating System was certainly a 10B application, perhaps worth 100B.

nanolith 3y ago

Dijkstra grappled with these issues. Two classic notes of his are: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/EWD303.html https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/E... https://www.cs.utexas.edu/~EWD/transcriptions/EWD10xx/EWD1036.html https://www.cs.utexas.edu/~EWD/transcriptions/EWD10xx/EWD103... There are several takeaways here, but three of my favorite highlights: * Dijkstra actually did not like calling such errors "bugs", as it is a framing problem. * Likewise, he believed that we should avoid anthropomorphizing software or identifying with it. * Finally, he believed in building a formal specification of software and then proving that the software matched this specification. Part of this was to avoid "bug fixing" hunts in which the focus was on "debugging" instead of correctness. But also, this was to ensure that there was a proper system view of the software, which ties back into the conclusion of the article here.

rramadass 3y ago
Nice! It is a tragedy that programmers nowadays don't read Dijkstra nor think about what he wrote and meant. I tell people to think of him as their "Cus D'Amato" if they want to be a "Mike Tyson" in their field i.e. a man who has thought deeply about the subject, knows all the angles and can "train" one in the "correct" manner of writing programs.
shiandow 3y ago
Interestingly most bugs I encounter are bugs in the specification, so in some ways formal specification just moves the problem.
- nanolith 3y ago
  Formal specification means building up this specification from theory, and building proofs along the way. Of course, there can always be errors in specification, but this is supposed to be the first place we implement SAT solvers or proof assistants. During Dijkstra's time, such technologies were not yet available as they are now. There is work to be done until this is all practical and the overhead to do this is within the current overhead of software engineering, but we are quickly reaching that point in time.

eviks 3y ago

Don't understand why this can't be called a bug > The software runs slowly because large amounts of unnecessary data are being sent to the users.

regularfry 3y ago
Depends if it's a consequence of intentional design or not. It's useful to have a separate designation for accidents of implementation.
- boxed 3y ago
  "Unnecessary" was the word in the blog post. That word implies it can be just straight up fixed.
  - regularfry 3y ago
    Not necessarily. "Unnecessary" to the client doesn't mean that the server can know that ahead of sending it. A redesign can be the only way to get out of that.

ChrisMarshallNY 3y ago

Reminds me of McConnell’s Software Quality at Top Speed essay, from almost 30 years ago[0]. This chart, in particular[1]. These issues are fundamental, and persist, regardless of the technology, or state of the art. plus ça change [0] https://stevemcconnell.com/articles/software-quality-at-top-speed/ https://stevemcconnell.com/articles/software-quality-at-top-... [1] https://stevemcconnell.com/wp-content/uploads/2017/08/art04-1.gif https://stevemcconnell.com/wp-content/uploads/2017/08/art04-...

pfdietz 3y ago

Also, even if you just look at bugs, they will level off at some equilibrium level if the software is being changed (beyond fixing bugs), since change introduces bugs.

hasoleju 3y ago

There is always a tradeoff between different types of use cases. Some users solve problem A with the sofware. An other group solves problem B. Using the same software in both cases leads to edge cases that are bugs for one group but cannot be fixed without introducing bugs for the second group. The clear specification really is the solution to resolve this issue. That you don't declare behavior as buggy that is according to specification.

jfil 3y ago
You may hit a point where the software can't balance the needs of problem A and problem B users. Then, the solution is to split the product into 2 distinct ones. Or to exit the market for problem A in order to focus on delivering a solid solution for problem B.

w10-1 3y ago

It's not really bugs vs design issues. It's more changeable vs unchangeable. Part of that is technical cost, but in my experience most is perceived risk, which is largely a function of understanding and team (leader) confidence. The scariest things are existential: telling people the system they believe in has fatal flaws introduces a gap that can only be filled with a lot of confidence, which is typically lacking when growing fast or with new systems. The main remedy is to be sure you have an actual view and understanding of the system, and address things from that perspective. Otherwise, bugs are just make-work.