5 ms·

When Frank Marshall was hired as VP of engineering at Cisco in 1992, he assessed after three months that we had a software quality problem and forbade any softw

When Frank Marshall was hired as VP of engineering at Cisco in 1992, he assessed after three months that we had a software quality problem and forbade any software engineering activity except "Fixing bugs." This went on for perhaps six months. It allowed key competitors to get ahead of us because we could not address some serious architectural issues in the code base because that would be writing new code, not fixing bugs. Gerald Weinberg talks about "Software Stalins," managers who grab onto one indicator and think that driving it to zero (or 100, or 11 for you Spinal Tap Fans) will resolve all other problems. If only things were so simple.

throwiforgtnlzy 3y ago
The lack of nuance to consider exceptions appears to have been the problem. Perhaps a lot of products needed bug fix campaigns because most corporations tend to reward SWE productivity based on so-called "evidence" like: - KLoC - # of commits - adding features - so-called "impact" , rather than: - reducing the cost to maintain code - making code more robust - making features actually work as users expect - rip-replacing what shouldn't be salvaged because it's too expensive So instead of adopting a funny mustache and making inflexible commandments, it would be less disruptive to have targeted bug fix campaigns where needed to boost nonfunctional requirements like quality, reducing support costs, and improving user satisfaction to their desired target levels.
swatcoder 3y ago
This is a really interesting anecdote because (obviously) Cisco fared quite well on net over the last severl decades, is largely respected as a reliable vendor for critical infrastructure, and doesn't evidence even a hiccup in market assessments of its health during that time. https://www.macrotrends.net/stocks/charts/CSCO/cisco/stock-price-history https://www.macrotrends.net/stocks/charts/CSCO/cisco/stock-p... I don't mean to take away the truth of your experience or dismiss that you might better know the disconnect between what you describe and what outsiders might see in hindsight -- it's just the disconnect itself that's fascinating. > If only things were so simple. Indeed!
- skmurphy 3y ago
  The renewed growth and increasing customer satisfaction--and our long-term success--were due to engineering abandoning a poorly thought-out policy before it crippled us instead of merely hampering us. Part of the challenge in a "freeze spec" approach implicit in "only fix bugs" is that it assumes the market requirements are well established. That was definitely not the case for Internet protocols/functionality in 1992. HTTP traffic did not exist yet (along with a couple of dozen other things that would quickly become dominant by 1997. The effect of the bug fix stall was to give our more nimble competitors more growth and more runway. The fact that Cisco did very well was despite the "just fix bugs" mandate. I offered the story because I thought some folks reading the linked post would find it unbelievable that a company would decide to fixate on a single simple metric when operating in a complex and evolving environment.
  - kazinator 3y ago
    You have to do both, on different branches. Rearchitect things in one branch, fix nothing but bugs in another. Forward port all the bugfixes you can from one to the other. Though: version control ... 1992 ... haha. Hey, that's when that unwieldy behemoth ClearCase was released.
    - pjc50 3y ago
      I think that approach would be doomed even in the present day. Multiple "legacy" release branches is very hard to manage. Only the very largest such as Microsoft can do it, and they clearly hate doing so and want to push to rolling releases like everyone else.
      - worthless-trash 3y ago
        Red Hat / RHEL does this on all EUS release every day, on RHEL 6,7,8,9. Tell me more about how you would subscribe to a MS "bugfix only channel of updates" ? This is my first time hearing about that.
shiroiushi 3y ago
It's too bad that Microsoft didn't take this same approach, and stick with it for a long time. I'm wondering what things are like in the alternate universe where OS/2 became the dominant PC OS instead of Windows.
sgift 3y ago
Architectural problems are bugs. Hopefully, most companies have VP of engineering who understand that.
- Attrecomet 3y ago
  Disagree. Architecture problems are different enough from software bugs that it's worth it to keep them separate words.
  - chuckadams 3y ago
    If bad architecture makes the system fall over, that's a bug. It's not black and white.
IAmYourDensity 3y ago
> "Software Stalins," managers who grab onto one indicator and think that driving it to zero (or 100, or 11 for you Spinal Tap Fans) will resolve all other problems. Paul O’Neill famously did exactly this at Alcoa starting in 1987, focusing solely on worker safety resolved many other problems and multiplied profitability. >The company's market value increased from $3 billion in 1986 to $27.53 billion in 2000, while net income increased from $200 million to $1.484 billion. [0] https://www.forbes.com/sites/roddwagner/2019/01/22/have-we-learned-the-alcoa-keystone-habit-lesson/ https://www.forbes.com/sites/roddwagner/2019/01/22/have-we-l...
- skmurphy 3y ago
  It's a reasonable counter-example, two other CEOs of successful firms who emphasized safety were Lawrence Culp at Danaher (he is now at GE) and David Cote at Honeywell. All three had tenures more than a decade where their firms invested in and introduced new products, so I suspect they focused on more than worker safety but I am not personally familiar with their policies and decisions. I think it makes a very good story that a focus on worker safety--to the exclusion of any other objectives--is all that you need. But I don't think anything is that simple.

throwiforgtnlzy 3y ago

The lack of nuance to consider exceptions appears to have been the problem. Perhaps a lot of products needed bug fix campaigns because most corporations tend to reward SWE productivity based on so-called "evidence" like: - KLoC - # of commits - adding features - so-called "impact" , rather than: - reducing the cost to maintain code - making code more robust - making features actually work as users expect - rip-replacing what shouldn't be salvaged because it's too expensive So instead of adopting a funny mustache and making inflexible commandments, it would be less disruptive to have targeted bug fix campaigns where needed to boost nonfunctional requirements like quality, reducing support costs, and improving user satisfaction to their desired target levels.

swatcoder 3y ago

This is a really interesting anecdote because (obviously) Cisco fared quite well on net over the last severl decades, is largely respected as a reliable vendor for critical infrastructure, and doesn't evidence even a hiccup in market assessments of its health during that time. https://www.macrotrends.net/stocks/charts/CSCO/cisco/stock-price-history https://www.macrotrends.net/stocks/charts/CSCO/cisco/stock-p... I don't mean to take away the truth of your experience or dismiss that you might better know the disconnect between what you describe and what outsiders might see in hindsight -- it's just the disconnect itself that's fascinating. > If only things were so simple. Indeed!

skmurphy 3y ago
The renewed growth and increasing customer satisfaction--and our long-term success--were due to engineering abandoning a poorly thought-out policy before it crippled us instead of merely hampering us. Part of the challenge in a "freeze spec" approach implicit in "only fix bugs" is that it assumes the market requirements are well established. That was definitely not the case for Internet protocols/functionality in 1992. HTTP traffic did not exist yet (along with a couple of dozen other things that would quickly become dominant by 1997. The effect of the bug fix stall was to give our more nimble competitors more growth and more runway. The fact that Cisco did very well was despite the "just fix bugs" mandate. I offered the story because I thought some folks reading the linked post would find it unbelievable that a company would decide to fixate on a single simple metric when operating in a complex and evolving environment.
- kazinator 3y ago
  You have to do both, on different branches. Rearchitect things in one branch, fix nothing but bugs in another. Forward port all the bugfixes you can from one to the other. Though: version control ... 1992 ... haha. Hey, that's when that unwieldy behemoth ClearCase was released.
  - pjc50 3y ago
    I think that approach would be doomed even in the present day. Multiple "legacy" release branches is very hard to manage. Only the very largest such as Microsoft can do it, and they clearly hate doing so and want to push to rolling releases like everyone else.
    - worthless-trash 3y ago
      Red Hat / RHEL does this on all EUS release every day, on RHEL 6,7,8,9. Tell me more about how you would subscribe to a MS "bugfix only channel of updates" ? This is my first time hearing about that.

shiroiushi 3y ago

It's too bad that Microsoft didn't take this same approach, and stick with it for a long time. I'm wondering what things are like in the alternate universe where OS/2 became the dominant PC OS instead of Windows.

sgift 3y ago

Architectural problems are bugs. Hopefully, most companies have VP of engineering who understand that.

Attrecomet 3y ago
Disagree. Architecture problems are different enough from software bugs that it's worth it to keep them separate words.
- chuckadams 3y ago
  If bad architecture makes the system fall over, that's a bug. It's not black and white.

IAmYourDensity 3y ago

> "Software Stalins," managers who grab onto one indicator and think that driving it to zero (or 100, or 11 for you Spinal Tap Fans) will resolve all other problems. Paul O’Neill famously did exactly this at Alcoa starting in 1987, focusing solely on worker safety resolved many other problems and multiplied profitability. >The company's market value increased from $3 billion in 1986 to $27.53 billion in 2000, while net income increased from $200 million to $1.484 billion. [0] https://www.forbes.com/sites/roddwagner/2019/01/22/have-we-learned-the-alcoa-keystone-habit-lesson/ https://www.forbes.com/sites/roddwagner/2019/01/22/have-we-l...

skmurphy 3y ago
It's a reasonable counter-example, two other CEOs of successful firms who emphasized safety were Lawrence Culp at Danaher (he is now at GE) and David Cote at Honeywell. All three had tenures more than a decade where their firms invested in and introduced new products, so I suspect they focused on more than worker safety but I am not personally familiar with their policies and decisions. I think it makes a very good story that a focus on worker safety--to the exclusion of any other objectives--is all that you need. But I don't think anything is that simple.